From 2060dc815c30e04d199b363d791bb67975b68537 Mon Sep 17 00:00:00 2001 From: Leo Vasanko Date: Fri, 28 Aug 2026 14:57:04 +0000 Subject: [PATCH] Per-hostname data directory /{content.kantadb, analytics.json, files} from CLI arg; drop learned site_url. The first positional CLI argument (default localhost) names the site's public hostname and its data directory under the cwd, replacing the CWD-relative pagerite.* files. The public origin (https://) is now authoritative configuration instead of a value learned from admin browsers: POST /_api/site-url, Data.site_url and the pagerite.js reporter are removed, and page rendering, sitemap, robots.txt and analytics own_origin all use SITE_URL consistently (localhost falls back to the request's base URL). --- .gitignore | 3 +- AGENTS.md | 1 + docs/analytics.md | 20 ++- docs/backend.md | 4 +- docs/content-model.md | 6 +- docs/design-principles.md | 2 +- docs/frontend-runtime.md | 8 +- frontend/src/AnalyticsView.vue | 7 +- frontend/src/TrailLink.vue | 14 ++ frontend/src/TransitionGraph.vue | 25 +++- frontend/src/analytics/format.js | 10 ++ frontend/src/pagerite.js | 8 -- pagerite/__main__.py | 11 ++ pagerite/analytics.py | 55 ++++++++ pagerite/app.py | 216 +++++++++++++++++++++++-------- pagerite/data.py | 15 +-- pagerite/migrations.py | 23 ++++ 17 files changed, 334 insertions(+), 94 deletions(-) create mode 100644 pagerite/migrations.py diff --git a/.gitignore b/.gitignore index 9a83443..f0f3254 100644 --- a/.gitignore +++ b/.gitignore @@ -1,8 +1,7 @@ .* !.gitignore *.lock -*.kantadb -pagerite.analytics.json +/localhost dbip-*.mmdb* /pagerite/frontend-build package-lock.json diff --git a/AGENTS.md b/AGENTS.md index b451ef1..3f89914 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke - `pagerite/` — Python backend package (hatchling build target). - `app.py` — FastAPI app and route registration. - `data.py` — msgspec Structs for the kanta database. + - `migrations.py` — kanta schema migrations (`migrate_vN`), e.g. v1 moves legacy in-db file blobs to the on-disk store. - `markdown.py` — markdown-it-py renderer. - `views.py` — shared page layout and rendering. - `seed.py` — demo content, written only on first database creation. diff --git a/docs/analytics.md b/docs/analytics.md index 27a4576..0e03e04 100644 --- a/docs/analytics.md +++ b/docs/analytics.md @@ -2,11 +2,11 @@ Server-side visit analytics. Data lives in a plain JSON file — a msgspec Struct dumped to disk — separate from the kanta content database, path from -`PAGERITE_ANALYTICS` (default: the database path with `.kantadb` replaced by -`.analytics.json`, e.g. `pagerite.analytics.json`). +`PAGERITE_ANALYTICS` (default: `analytics.json` in the per-site data +directory, e.g. `localhost/analytics.json`). - `pagerite/analytics.py` — data model (`Analytics`, `Client`, `Visit`, - `CrawlerHit`, `AbuseHit`) and the `Store` (in-memory data + session map, + `CrawlerHit`, `AbuseHit`, `Favicon`) and the `Store` (in-memory data + session map, atomic JSON persistence). - `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`), the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics` @@ -73,6 +73,20 @@ falsy values are omitted): - The server validates `to`: internal paths must be valid slug paths ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the https origin and accepted only when the client sent exactly that. +- **External-site favicons**: for every external https origin seen as a visit + referer or an exit link, the server fetches `{origin}/favicon.ico` in a + background task (httpx, 8 s timeout, ≤ 64 KB, image content-types only — + SVG is sniffed from the body when served without an image type) and stores + the icon content-hashed on disk in the FileStore (served at `/_f/{name}`, + extension matching the actual MIME). The origin → file name mapping is + recorded in `Analytics.favicons` (`Favicon.file`/`fetched`); misses are + recorded too and retried only after 7 days. Fetches are scheduled after + each ping and once at startup, which backfills icons for already-recorded + data. The viewer payload carries `favicons` (origin → `/_f/...` path), + and the viewer shows the icon wherever an external site is mentioned: + referer/exit trail links in the visit table and the source/exit pills of + the transition map (UTM-attributed source nodes without an https origin + stay text-only). - **Client records**: the visitor's IP (IPv4 or IPv6 /64 network), raw `User-Agent` and extracted `Accept-Language` tag are hashed with blake3; the first 6 bytes identify a shared `Client` record. The `Client` stores diff --git a/docs/backend.md b/docs/backend.md index 0b1e90d..b850431 100644 --- a/docs/backend.md +++ b/docs/backend.md @@ -10,6 +10,8 @@ The build mirrors the URL space — hashed immutable assets under `/_assets/`, ` Generated HTML pages (content pages, category/404 placeholders, `/_a`) go through `_html_response`: zstd-compressed per request at level 9 when the client sends `accept-encoding: zstd` (no gzip fallback; static assets are pre-compressed by the `Frontend`), with `vary: accept-encoding` set and the ETag kept identical across encodings so `if-none-match` revalidation still works. In production the rendered bodies are cached in an LRU keyed by everything the output depends on — page kind, path, the site origin (social meta), encoding, and `data.version`, which bumps on every content/settings change and so transparently invalidates the whole cache. The cache is bypassed in dev, where theme/design CSS is re-read from disk per request. Content pages carry an ETag built from the node's modified timestamp and `data.version`; `/_a` instead gets a blake3 hash of the rendered body (it has no Node), with matching `if-none-match` revalidations answered by a 304. +Uploaded files, seed assets and fetched external-site favicons live in the `FileStore`: content-addressed files on disk under `/files/` (`PAGERITE_FILES`), fully cached in RAM at startup — both the raw body and a zstd-compressed copy (kept only when smaller). `GET /_f/{name}` serves from the RAM cache with immutable caching, answering the zstd variant when the client accepts it; the name is the ETag. Legacy databases that still carry blobs in a `files` kanta field are migrated to disk by `pagerite/migrations.py::migrate_v1` (kanta's `migrate_vN` mechanism, wired via `Kanta(..., migrations="pagerite.migrations")`), which pops the field from the raw state before struct decoding. + ## `data.py` msgspec Structs for the kanta database. See `docs/content-model.md` for the full data model. @@ -24,7 +26,7 @@ markdown-it-py renderer (html passthrough + attrs, footnote, deflist, tasklists, The shared page layout as an html5tagger `Template` with placeholders (`Title`, `Brand`, `Banner`, `Nav`, `Sidebar`, `Main`), nav rendering straight from the `Data.menu` tree (siblings sorted by `Node.order`; nav links to content-less labels point at their first child via `first_leaf`, the first published descendant with content), and page/404 rendering. -Content pages get SEO/social meta (description, canonical link, Open Graph + twitter card) from heuristics over the rendered article: the description is the first paragraph's text, the share image prefers a `{.hero}`-classed image, then the first raster ``, then the first SVG; the first `