diff --git a/.gitignore b/.gitignore index 9a83443..f0f3254 100644 --- a/.gitignore +++ b/.gitignore @@ -1,8 +1,7 @@ .* !.gitignore *.lock -*.kantadb -pagerite.analytics.json +/localhost dbip-*.mmdb* /pagerite/frontend-build package-lock.json diff --git a/AGENTS.md b/AGENTS.md index b451ef1..3f89914 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke - `pagerite/` — Python backend package (hatchling build target). - `app.py` — FastAPI app and route registration. - `data.py` — msgspec Structs for the kanta database. + - `migrations.py` — kanta schema migrations (`migrate_vN`), e.g. v1 moves legacy in-db file blobs to the on-disk store. - `markdown.py` — markdown-it-py renderer. - `views.py` — shared page layout and rendering. - `seed.py` — demo content, written only on first database creation. diff --git a/docs/analytics.md b/docs/analytics.md index 27a4576..0e03e04 100644 --- a/docs/analytics.md +++ b/docs/analytics.md @@ -2,11 +2,11 @@ Server-side visit analytics. Data lives in a plain JSON file — a msgspec Struct dumped to disk — separate from the kanta content database, path from -`PAGERITE_ANALYTICS` (default: the database path with `.kantadb` replaced by -`.analytics.json`, e.g. `pagerite.analytics.json`). +`PAGERITE_ANALYTICS` (default: `analytics.json` in the per-site data +directory, e.g. `localhost/analytics.json`). - `pagerite/analytics.py` — data model (`Analytics`, `Client`, `Visit`, - `CrawlerHit`, `AbuseHit`) and the `Store` (in-memory data + session map, + `CrawlerHit`, `AbuseHit`, `Favicon`) and the `Store` (in-memory data + session map, atomic JSON persistence). - `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`), the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics` @@ -73,6 +73,20 @@ falsy values are omitted): - The server validates `to`: internal paths must be valid slug paths ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the https origin and accepted only when the client sent exactly that. +- **External-site favicons**: for every external https origin seen as a visit + referer or an exit link, the server fetches `{origin}/favicon.ico` in a + background task (httpx, 8 s timeout, ≤ 64 KB, image content-types only — + SVG is sniffed from the body when served without an image type) and stores + the icon content-hashed on disk in the FileStore (served at `/_f/{name}`, + extension matching the actual MIME). The origin → file name mapping is + recorded in `Analytics.favicons` (`Favicon.file`/`fetched`); misses are + recorded too and retried only after 7 days. Fetches are scheduled after + each ping and once at startup, which backfills icons for already-recorded + data. The viewer payload carries `favicons` (origin → `/_f/...` path), + and the viewer shows the icon wherever an external site is mentioned: + referer/exit trail links in the visit table and the source/exit pills of + the transition map (UTM-attributed source nodes without an https origin + stay text-only). - **Client records**: the visitor's IP (IPv4 or IPv6 /64 network), raw `User-Agent` and extracted `Accept-Language` tag are hashed with blake3; the first 6 bytes identify a shared `Client` record. The `Client` stores diff --git a/docs/backend.md b/docs/backend.md index 0b1e90d..b850431 100644 --- a/docs/backend.md +++ b/docs/backend.md @@ -10,6 +10,8 @@ The build mirrors the URL space — hashed immutable assets under `/_assets/`, ` Generated HTML pages (content pages, category/404 placeholders, `/_a`) go through `_html_response`: zstd-compressed per request at level 9 when the client sends `accept-encoding: zstd` (no gzip fallback; static assets are pre-compressed by the `Frontend`), with `vary: accept-encoding` set and the ETag kept identical across encodings so `if-none-match` revalidation still works. In production the rendered bodies are cached in an LRU keyed by everything the output depends on — page kind, path, the site origin (social meta), encoding, and `data.version`, which bumps on every content/settings change and so transparently invalidates the whole cache. The cache is bypassed in dev, where theme/design CSS is re-read from disk per request. Content pages carry an ETag built from the node's modified timestamp and `data.version`; `/_a` instead gets a blake3 hash of the rendered body (it has no Node), with matching `if-none-match` revalidations answered by a 304. +Uploaded files, seed assets and fetched external-site favicons live in the `FileStore`: content-addressed files on disk under `/files/` (`PAGERITE_FILES`), fully cached in RAM at startup — both the raw body and a zstd-compressed copy (kept only when smaller). `GET /_f/{name}` serves from the RAM cache with immutable caching, answering the zstd variant when the client accepts it; the name is the ETag. Legacy databases that still carry blobs in a `files` kanta field are migrated to disk by `pagerite/migrations.py::migrate_v1` (kanta's `migrate_vN` mechanism, wired via `Kanta(..., migrations="pagerite.migrations")`), which pops the field from the raw state before struct decoding. + ## `data.py` msgspec Structs for the kanta database. See `docs/content-model.md` for the full data model. @@ -24,7 +26,7 @@ markdown-it-py renderer (html passthrough + attrs, footnote, deflist, tasklists, The shared page layout as an html5tagger `Template` with placeholders (`Title`, `Brand`, `Banner`, `Nav`, `Sidebar`, `Main`), nav rendering straight from the `Data.menu` tree (siblings sorted by `Node.order`; nav links to content-less labels point at their first child via `first_leaf`, the first published descendant with content), and page/404 rendering. -Content pages get SEO/social meta (description, canonical link, Open Graph + twitter card) from heuristics over the rendered article: the description is the first paragraph's text, the share image prefers a `{.hero}`-classed image, then the first raster ``, then the first SVG; the first `