pagerite/docs/analytics.md

Analytics

Server-side visit analytics. Data lives in a plain JSON file — a msgspec Struct dumped to disk — separate from the kanta content database, path from PAGERITE_ANALYTICS (default: the database path with .kantadb replaced by .analytics.json, e.g. pagerite.analytics.json).

• pagerite/analytics.py — data model (Analytics, Visit) and the Store (in-memory data + session map, atomic JSON persistence).
• pagerite/app.py — entry-referer stashing in show_page (_track_entry), the POST /_a ping endpoint, and WebSocket /_api/ws/analytics (admin-gated like every /_api endpoint).
• frontend/src/pagerite.js — client navigation pings and the 📊 pen.
• frontend/src/AnalyticsView.vue — viewer component rendered inside the normal site layout on the /_a analytics page.
• frontend/src/analytics-main.js — page entry that mounts AnalyticsView into #analytics-app inside #main.

What is collected

The client (pagerite.js) POSTs fire-and-forget pings to /_a with {fr, to} (fr = source path):

• Initial page load: to is the loaded path. This ping is what starts the visit and counts the entry page view — the document GET alone records nothing, so bots and admin browsing never register. Reloads are not visits: the ping is skipped (PerformanceNavigationTiming reload), so a refresh neither counts a second view nor logs a self-transition. The GET handler stashes a cross-origin https Referer (origin part only) and any utm_* query parameters in in-memory IP tables, consumed by the ping that starts the visit; internal or absent referers never touch the referer table.
• Internal fetch-navigations: to is the target path, sent only after the swap actually happened (a failed swap falls back to a full load, whose initial ping counts the view instead — no gap, no double count).
• External links (https only): to is the link's origin. This is the exit-link record; the user may continue navigating afterwards (new tab, back), so the exit origin is not necessarily the last trail entry.
• Excluded: back/forward (popstate) navigations, navigation involving the analytics page itself (/_a), and everything while the user is known to be an admin and SSO is actually in use — with no auth proxy (dev/test) "admin" is everyone's state, so the gate is off and everything is recorded — or has the editor open (body.editing). Admin noise, not visits.
• The server validates to: internal paths must be valid slug paths ("/" or [a-z0-9_-] segments), external ones are re-derived to the https origin and accepted only when the client sent exactly that.
• The initial ping also records the visitor's User-Agent and Accept-Language headers. The first Accept-Language tag is stored as lang (e.g. en-us) and its region subtag, if present, is stored as an initial country (e.g. US).
• The visitor IP is stored. A reverse-DNS lookup is attempted for each new visit and the result, when available, is cached in RAM and stored as host; local/reserved/multicast addresses are skipped.
• If a DB-IP MMDB file (dbip-*.mmdb or dbip-*.mmdb.gz) is present in the repository root, it is loaded at startup and used to look up a more accurate country. The MMDB lookup and the reverse-DNS lookup run in background tasks after the visit is stored, so the / _a response is never delayed. The decompressed dbip-*.mmdb file is kept in the repository root and ignored by git.
• Crawler hits: every document GET is queued in RAM as a pending crawler hit. If a ping from the same (IP, User-Agent) pair arrives within 10 seconds the hit is discarded; otherwise it is written to crawlers. Crawlers do not count as visits or views. In the analytics viewer, crawler hits are grouped by the same (IP, User-Agent) pair and shown as a trail of internal pages that crawler visited; the crawler table lists the most active crawlers first rather than the most recent hits.

Visits and sessions

There are no cookies. A visit is tied together by the (IP, User-Agent) pair (IP from the first X-Forwarded-For hop — we sit behind a proxy — else the direct peer): the first ping from a pair starts a new visit, subsequent pings extend it. Pings arriving with no known session (server restart) start a fresh visit from the first ping — treated as missing data rather than dropped. The (IP, UA) → visit map and the IP → entry-referer/UTM tables are in-memory only, but the IP and any resolvable reverse-DNS host name are stored on the Visit record itself.

Each Visit record:

• start — timestamp of the first event,
• entry — first page (path) seen,
• referer — external https origin of the initial load, "" for direct,
• ip — visitor IP address (first X-Forwarded-For hop, or direct peer),
• host — reverse-DNS host name for ip when resolvable, else "",
• trail — everything seen afterwards in first-seen order: page paths and external exit origins. Re-visiting an already seen page (incl. the entry) does not append.
• lang — first Accept-Language tag, lowercased (e.g. en-us),
• country — two-letter country code. Initially derived from the Accept-Language region subtag, but overwritten by the DB-IP MMDB result when a database is available,
• city — city name from the DB-IP MMDB lookup, when available,
• ua — raw User-Agent string from the initial ping,
• ua_pretty — compact display form of the UA (browser/OS/device) when parsable, otherwise the raw string,
• utm — utm_* query parameters from the landing URL, as a dict.

Each CrawlerHit record:

• start — timestamp of the document GET,
• entry — page path requested,
• ip — IP address,
• ua — raw User-Agent header,
• ua_pretty — compact display form of the UA when parsable,
• referer — external https origin of the request, "" for direct/none,
• query — raw query string of the request.

Crawler hits are grouped by User-Agent in the analytics viewer.

Aggregates

• transitions: time series of page transitions, sparse nested dict from -> to -> bucket -> count with the same 5-minute bucketing as views. from is the referer origin or "(direct)" for initial loads, a page path for pings.
• views: time series of page loads, path -> bucket -> count, sparse: only non-zero 5-minute buckets exist (bucket key is its floored ISO timestamp). Every load counts, including repeats within a visit; external exit origins are not page views and are not counted here.
• site_visits: bucket -> count of new visits started, same sparse 5-minute bucketing.

Sparseness keeps quiet sites small; dropping old data is a matter of deleting list/dict entries (visits is a plain append-only list, buckets plain keys).

Persistence

The whole Analytics struct is JSON-encoded and written atomically (temp file + rename) on every recorded event. Traffic on a small CMS makes this cheap enough; batching can be added later without changing the format.

Viewing

The 📊 pen in the banner corner (admins only, injected by pagerite.js next to the edit pens) links to /_a, the analytics page. It is a normal site page: the standard banner, navigation and footer stay in place, and the analytics content is rendered inside #main. The page itself is public, but the data stream comes from WebSocket /_api/ws/analytics, which remains admin-gated like the rest of the management API; visitors without access see the viewer with a "could not be loaded" message.

Because it is a real page, fetch-navigation handles it like any other internal link: clicking the 📊 pen (or any link to /_a) fetches the server-rendered HTML, swaps the dynamic regions and mounts the Vue analytics app in place. The range selector updates the URL query string (?range=week etc.) so links to a specific range can be shared.

AnalyticsView.vue is no longer a full-screen overlay; the body.analytics-open page-chrome hiding and #/analytics/<range> hash routing have been removed.

Charts are SVG curves (Catmull-Rom over an edge-aware adaptive Gaussian — a change-point detector splits the series at traffic-level shifts, then each segment is smoothed with a bandwidth that ramps with a broad pilot estimate of the local rate: isolated events stay narrow (~0.4-unit sigma, peaking at ~1 event/unit), busy traffic widens to a 1-unit sigma. The raw series is drawn faint underneath). Values are per-unit rates — per hour on the week view (5-minute bucket counts × 12, plotted at native 5-minute resolution), per day on the month+ ranges — and the smoothing time scale follows the unit: the month+ sigmas are 24× the hourly ones. The y max is derived from the smoothed curves so single-bucket spikes don't blow up the scale, and raw spikes are clamped into the plot. Axes always start at 0 and end at a multiple of a 1-2-5 major step (max 5 labeled intervals, minor lines at fifths when integral; the minimum y-axis range is 10 so tiny values such as a single visit are not stretched to a fractional scale). The week range is aligned to Monday 00:00 UTC and overlays up to 8 previous weeks in the same accent color at decreasing opacity (the current week is truncated at the current bucket, never drawing fake zeroes for the future); its x labels are weekday names centered at midday UTC, without vertical grid lines (day boundaries would be misleading in the viewer's timezone). The month view labels days the same lineless way — day numbers at noon UTC, with the month name substituted for the 1st. Year is a rolling 365-day window ending at now, re-bucketed to daily points, with boundary lines at months/years. All uses the full data reach, but keeps at least the past 30 days so the chart never collapses to a tiny sliver when the site is young. Below the charts: a radial transition map (all pages from /_api/pages — front page at the center, each slug level on its own ring, siblings clockwise in navigation order from the top, radial gap equal to the arc spacing — opposite transition directions joined into organic tapered connections whose middle width grows logarithmically with the count (a single count renders as a ~1 px line, uncapped), connections carrying less than 1% of the total traffic pruned; beads are simulated one by one in JS (requestAnimationFrame) and flow along each edge, emitted at a rate linearly proportional to the directional count with no in-flight limit, opposing directions offset onto parallel lanes. External referers show as a node row above the map, external exits as small nodes fanned outwards from their source page), per-page view counts, the top transitions and the 50 most recent visit trails. Data is streamed live over WebSocket /_api/ws/analytics, which pushes the latest JSON snapshot on connect and again whenever the analytics file is updated (with a small server-side debounce to avoid flooding under high traffic).