From 31895065f8ff8cd144feb5df26d97b482751abbb Mon Sep 17 00:00:00 2001 From: Leo Vasanko Date: Sat, 29 Aug 2026 21:20:04 +0000 Subject: [PATCH] Collect analytics over a /_ws WebSocket instead of POST /_a pings --- docs/analytics.md | 75 +++++++++++------- frontend/src/pagerite.js | 165 ++++++++++++++++++++++++++------------- pagerite/analytics.py | 79 +++++++++++++------ pagerite/app.py | 83 +++++++++++--------- 4 files changed, 255 insertions(+), 147 deletions(-) diff --git a/docs/analytics.md b/docs/analytics.md index 0e03e04..310d896 100644 --- a/docs/analytics.md +++ b/docs/analytics.md @@ -9,9 +9,9 @@ directory, e.g. `localhost/analytics.json`). `CrawlerHit`, `AbuseHit`, `Favicon`) and the `Store` (in-memory data + session map, atomic JSON persistence). - `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`), - the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics` + the `/_ws` activity WebSocket, and `WebSocket /_api/ws/analytics` (admin-gated like every `/_api` endpoint). -- `frontend/src/pagerite.js` — client navigation pings and the 📊 pen. +- `frontend/src/pagerite.js` — the client activity channel and the 📊 pen. - `frontend/src/AnalyticsView.vue` — viewer component rendered inside the normal site layout on the `/_a` analytics page. - `frontend/src/analytics-main.js` — page entry that mounts `AnalyticsView` @@ -19,46 +19,61 @@ directory, e.g. `localhost/analytics.json`). ## What is collected -The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with -`fr`, `to`, `hide` and `read` as query parameters (`fr` = source path; -falsy values are omitted): +The client (`pagerite.js`) keeps a WebSocket connection to `/_ws` for the +whole browsing session and sends activity messages over it — JSON text +frames matching the server's `Ping` msgspec struct with the fields `fr` +(source path), `to` (navigation target), `read` (active seconds on `fr` +since the last report) and `hide`; falsy fields are omitted. One channel +follows the session, so the activity of a visit stays tied together, and +while the user is active the accumulated reading time is flushed every few +seconds: the trail times are cumulative, so a disconnection simply leaves +the last reported time in place (no close beacon). After 5 minutes without +any activity the client closes the socket itself — a sleeping browser tab +would lose it anyway — and the next activity reconnects as a fresh session; +reconnects are attempted only on user activity, with an exponential backoff +between attempts so a failing endpoint is never hammered. Idle-time link preloads +stay plain `fetch()` calls so the browser may cache the responses; the +WebSocket reports actual navigations and active time spent on a page. - **Initial page load**: only `to` — the loaded path — is sent, never `fr` (an `fr` equal to `to` would log a bogus self-transition when a session - already exists, e.g. a second tab). This ping is what starts + already exists, e.g. a second tab). This message is what starts the visit and counts the entry page view — the document GET alone records nothing, so bots never register (admin browsing does register, but flagged `hide`; see **Admins** below). JS-running crawlers - (Googlebot, GoogleOther, Applebot, ...) do ping, but their User-Agent - gives them away: pings whose UA matches `_is_bot_ua` (anything calling + (Googlebot, GoogleOther, Applebot, ...) do connect and report, but their + User-Agent gives them away: messages whose UA matches `_is_bot_ua` + (anything calling itself a "bot", plus known exceptions such as GoogleOther) are ignored server-side, and their document GETs land in the crawler list instead. No source-IP verification is done: a spoofed bot UA merely lands in the crawler stats, and scanners that probe telltale paths are caught by the abuse rules regardless. Reloads are not - visits: the ping is skipped (PerformanceNavigationTiming `reload`), so a + visits: the message is skipped (PerformanceNavigationTiming `reload`), so a refresh neither counts a second view nor logs a self-transition. The GET handler stashes a cross-origin https `Referer` (origin part only — unavailable to JS once the page has loaded) and any - `utm_*` query parameters in in-memory IP tables, consumed by the ping that + `utm_*` query parameters in in-memory IP tables, consumed by the first + message that starts the visit; internal or absent referers never touch the referer table. - **Internal fetch-navigations**: `to` is the target path, sent only after the swap actually happened (a failed swap falls back to a full load, - whose initial ping counts the view instead — no gap, no double count). + whose initial message counts the view instead — no gap, no double count). - **External links** (`https` only): `to` is the link's full URL. This is the exit-link record; the user may continue navigating afterwards (new tab, back), so the exit URL is not necessarily the last trail entry. Outbound links are stored by full URL so several links to the same domain remain distinct. - **Excluded**: back/forward (popstate) navigations, navigating *to* the - analytics page (`/_a` — its GET is untracked, and the server rejects it - as a ping target anyway), and everything while the user has the editor + analytics page (`/_a` — its GET is untracked, and the server cannot + record it as a navigation target anyway), and everything while the user has + the editor open (`body.editing`). Admin noise, not visits. Navigating *away* from - `/_a` does ping: the fetch-navigation already GET-ed the target page - without the preload header, and without the ping that GET would flush to + `/_a` does report: the fetch-navigation already GET-ed the target page + without the preload header, and without the message that GET would flush to the crawler list. - **Admins**: when SSO is in use and the session is known to be an admin, - the client still pings but adds `hide=1`. The activity is recorded as + the client still reports but adds `hide`. The activity is recorded as usual (navigations and all), but the `hide` flag is set on the **client record** — so it covers everything that client ever did: visits and crawler hits from before the login included. Hidden clients never appear @@ -81,7 +96,7 @@ falsy values are omitted): extension matching the actual MIME). The origin → file name mapping is recorded in `Analytics.favicons` (`Favicon.file`/`fetched`); misses are recorded too and retried only after 7 days. Fetches are scheduled after - each ping and once at startup, which backfills icons for already-recorded + each activity message and once at startup, which backfills icons for already-recorded data. The viewer payload carries `favicons` (origin → `/_f/...` path), and the viewer shows the icon wherever an external site is mentioned: referer/exit trail links in the visit table and the source/exit pills of @@ -100,8 +115,8 @@ falsy values are omitted): `host`; local/reserved/multicast addresses are skipped. If a DB-IP MMDB file (`dbip-*.mmdb` or `dbip-*.mmdb.gz`) is present in the repository root, it is loaded at startup and used to look up `country`/`city`. These - lookups run in background tasks after the event is stored, so the `/_a` - response is never delayed. The decompressed `dbip-*.mmdb` file is kept in + lookups run in background tasks after the event is stored, so WebSocket + message handling is never delayed. The decompressed `dbip-*.mmdb` file is kept in the repository root and ignored by git. The CLI flag `--dbip` (`uv run pagerite --dbip`) downloads the latest `dbip-city-lite-YYYY-MM.mmdb.gz` from DB-IP before the server starts, @@ -110,10 +125,11 @@ falsy values are omitted): file is used. - **Crawler hits**: every document GET is queued in RAM as a pending crawler hit — except idle-time link preloads from pagerite.js, which carry an - `x-pagerite-preload` header and are not tracked at all (the ping sent when - the user actually navigates to a preloaded page does the counting; forging + `x-pagerite-preload` header and are not tracked at all (the navigation + message sent when the user actually navigates to a preloaded page does + the counting; forging the header only hides a GET from the crawler stats, the path-based abuse - classification is unaffected). If a ping + classification is unaffected). If a message from the same client arrives within 10 seconds the hit is discarded; otherwise it is written to `crawlers` — unless the client is hidden (admin), in which case the hit is discarded on expiry too. Crawlers do not count as @@ -131,7 +147,7 @@ falsy values are omitted): random-UA scanner no longer pollutes the crawler stats of the legitimate bot it impersonates. Once classified, every document GET and 404 from the IP is recorded as an abuse hit with the full request path (query string - included), and its pings are ignored. The classified IP set (`abuse_ips`) + included), and its activity messages are ignored. The classified IP set (`abuse_ips`) is persisted in the JSON file; the plain-404 counters are RAM-only. In the viewer, abuse hits are grouped by IP (never by client/UA — scanners randomize theirs) in a separate "Abuse" table. Identical paths are @@ -145,9 +161,9 @@ falsy values are omitted): There are no cookies. A visit is tied together by a client hash — the first 6 bytes of a blake3 digest over the prettified IP (IPv4 unchanged, IPv6 /64 network), the raw `User-Agent` string and the extracted -`Accept-Language` tag. The first ping from a client hash starts a new -visit; subsequent pings extend it. Pings arriving with no known session -(server restart) start a fresh visit from the first ping — treated as +`Accept-Language` tag. The first message from a client hash starts a new +visit; subsequent messages extend it. Messages arriving with no known session +(server restart) start a fresh visit from the first message — treated as missing data rather than dropped. The client-hash → visit map and the IP → entry-referer/UTM tables are in-memory only; client metadata is stored in `Analytics.clients` keyed by the client hash. @@ -164,7 +180,7 @@ Each `Client` record: - `ua` — raw `User-Agent` string, - `ua_pretty` — compact display form of the UA (browser/OS/device) when parsable, otherwise the raw string, -- `hide` — true for admin clients (`hide=1` ping): all their visits, +- `hide` — true for admin clients (`hide` message field): all their visits, crawler hits and abuse hits are recorded but excluded from every statistic and from the viewer payload. @@ -180,7 +196,7 @@ Each `Visit` record: reading time in seconds (`read`) and the most recent HTTP status seen for the target (`status`). Re-visiting an already seen target updates its item instead of appending. -- `navs` — every navigation ping (`fr`, `to`), keyed by its timestamp, +- `navs` — every navigation message (`fr`, `to`), keyed by its timestamp, repeats included. The aggregates are computed from this log at display time. - `utm` — `utm_*` query parameters from the landing URL, as a dict. @@ -227,7 +243,8 @@ shapes, part of the WebSocket payload (`Display` struct alongside `visits`, - `transitions`: time series of page transitions, sparse nested dict `from -> to -> bucket -> count` with 5-minute bucketing. `from` is the - referer origin or `"(direct)"` for initial loads, a page path for pings. + referer origin or `"(direct)"` for initial loads, a page path for + navigations. - `views`: time series of page loads, `path -> bucket -> count`, sparse: only non-zero 5-minute buckets exist (bucket key is its floored ISO timestamp). Every load counts, including repeats within a visit; external exit origins diff --git a/frontend/src/pagerite.js b/frontend/src/pagerite.js index f7c5315..24ce310 100644 --- a/frontend/src/pagerite.js +++ b/frontend/src/pagerite.js @@ -347,8 +347,8 @@ import "overlayscrollbars/overlayscrollbars.css"; for (const url of urls) { if (pageCache.has(url)) continue; // x-pagerite-preload: idle cache warm-up, not a page view — the - // server excludes these GETs from analytics (the ping sent on actual - // navigation does the counting). + // server excludes these GETs from analytics (the navigation message + // sent on actual navigation does the counting). fetch(url, { headers: { "x-pagerite-preload": "1" } }) .then((r) => (r.ok && (r.headers.get("content-type") || "").includes("text/html") ? r.text() : "")) @@ -432,61 +432,111 @@ import "overlayscrollbars/overlayscrollbars.css"; }); }, { passive: true }); - // --- Analytics pings --------------------------------------------------- - // Fire-and-forget POSTs to /_a with the fields as query parameters (a - // beacon can carry no body, and query args show in server logs next to - // the document GET they refer to): on the initial page load (starts the - // visit — the server counts nothing from the document GET alone), for - // internal fetch-navigations, for external https exits, and on window - // close. ``read`` is the active time (ms) spent on ``fr``. - // Reading time pauses after 1 minute of inactivity and resumes on the - // next mouse/touch/scroll/keyboard event. - // Excluded: back/forward (popstate never pings), everything while the + // --- Analytics over WebSocket ------------------------------------------ + // One /_ws connection follows the whole browsing session: the initial page + // load (starts the visit — the server counts nothing from the document GET + // alone), internal fetch-navigations, external https exits, and frequent + // active reading-time updates. Messages are JSON text frames matching the + // server's msgspec Ping struct: {fr?, to?, read?, hide?} — falsy fields + // are omitted. ``read`` is the active time (ms) accumulated on ``fr`` since + // the last report; reading time pauses after 1 minute of inactivity and + // resumes on the next mouse/touch/scroll/keyboard event. While the user is + // active, accumulated reading time is flushed every few seconds, so a + // disconnect simply leaves the last reported time on the server — no close + // beacon is needed. After 5 minutes without any activity the client closes + // the channel itself (a sleeping tab would lose it anyway); the next + // activity reconnects and the server sees a new session. + // Excluded: back/forward (popstate never reports), everything while the // editor is open (body.editing — admin noise, not visits), and - // navigations TO the analytics page (/_a — admin machinery, and the - // server rejects it as a ping target anyway). Navigations AWAY from /_a - // must ping: load() already fetched the target page without the preload - // header, and without the ping that GET would flush to the crawler list. + // navigations TO the analytics page (/_a — admin machinery). Navigations + // AWAY from /_a must report: load() already fetched the target page + // without the preload header, and without the message that GET would flush + // to the crawler list. // Admins (when SSO is actually in use — with no auth proxy "admin" is - // everyone's state) ping normally but with hide=1: the server then - // records nothing and scrubs any session the same browser accumulated - // before logging in, so admins never show up as visits or crawlers. + // everyone's state) report normally but with hide: the server then flags + // the client record, scrubbing everything it ever did from the statistics, + // so admins never show up as visits or crawlers. // See docs/analytics.md. - // fetch wrapper: every key of ``params`` becomes a query arg on /_a - // (falsy values are omitted). Admins get hide=1. ``beacon`` uses - // sendBeacon when available, for unload-time pings. - function pingFetch(params, { beacon = false } = {}) { - const query = new URLSearchParams(); - if (ssoAvailable && isAdmin) params = { ...params, hide: 1 }; - for (const [key, value] of Object.entries(params)) { - if (value) query.set(key, value); - } - const url = `/_a?${query}`; + // The activity WebSocket. Messages sent before the connection opens are + // queued (the queue keeps the interim activity). Reconnects are driven by + // user activity only — never by timers while the page sits idle — with an + // exponential falloff between attempts so a failing endpoint cannot make + // us hammer the server (or trip its security limits). After a longer + // stretch without any activity we close the socket proactively: the user + // has moved on and left the tab open (a sleeping browser tab would lose + // the connection anyway), so the next activity reconnects and registers + // as a fresh session. Analytics must never break navigation: every send + // is wrapped, and a server without the endpoint just leaves the socket + // failing in the background. + let ws = null; + const wsQueue = []; + let wsReconnectMs = 1000; + let wsNotBefore = 0; + + function activityWs() { + if (ws || Date.now() < wsNotBefore) return; + const url = new URL("/_ws", location.href); + url.protocol = url.protocol === "https:" ? "wss:" : "ws:"; try { - if (beacon && navigator.sendBeacon) { - navigator.sendBeacon(url); - } else { - fetch(url, { method: "POST", keepalive: true }); - } - } catch { /* analytics must never break navigation */ } + ws = new WebSocket(url); + } catch { + return; + } + ws.onopen = () => { + wsReconnectMs = 1000; + for (const msg of wsQueue.splice(0)) ws.send(JSON.stringify(msg)); + }; + ws.onclose = () => { + ws = null; + // No timer here: the next user activity retries, after the backoff. + wsNotBefore = Date.now() + wsReconnectMs; + wsReconnectMs = Math.min(wsReconnectMs * 2, 30_000); + }; + ws.onerror = () => ws.close(); } - function ping({ to, fr = currentPath, read = 0, beacon = false } = {}) { + function report(msg) { if (document.body.classList.contains("editing")) return; - if (to === "/_a") return; - pingFetch({ fr, to, read: Math.round(read / 1000) }, { beacon }); + if (msg.to === "/_a") return; + if (ssoAvailable && isAdmin) msg.hide = true; + activityWs(); + if (ws?.readyState === WebSocket.OPEN) { + try { + ws.send(JSON.stringify(msg)); + return; + } catch { /* fall through to queueing */ } + } + wsQueue.push(msg); + } + + function ping({ to, fr = currentPath, read = 0 } = {}) { + // Reading-time updates from the analytics page itself are not tracked + // (/_a is admin machinery; the server would reject the path anyway). + if (!to && currentPath === "/_a") return; + const msg = {}; + if (fr) msg.fr = fr; + if (to) msg.to = to; + const secs = Math.round(read / 1000); + if (secs > 0) msg.read = secs; + if (!msg.to && !msg.read) return; + report(msg); } // Active reading time for the current page. The clock stops after 1 minute // without activity and restarts on the next mouse/touch/scroll/keyboard - // event. + // event. Every READ_FLUSH_MS of accumulated activity is reported. After + // IDLE_MS with no activity at all, the remaining read time is flushed and + // the WebSocket is closed: the user has moved on, and the next activity + // reconnects as a new session. const INACTIVE_MS = 60_000; + const READ_FLUSH_MS = 5_000; + const IDLE_MS = 5 * 60_000; let readStart = performance.now(); let readElapsed = 0; let reading = true; let readInactivityTimer = null; - let closePingedFor = null; + let idleTimer = null; function markReadActivity() { if (!reading) { @@ -500,6 +550,24 @@ import "overlayscrollbars/overlayscrollbars.css"; reading = false; } }, INACTIVE_MS); + // Any activity is a sign of life: (re)connect the channel if it was + // dropped or idle-closed (not while editing — admin noise), and push + // the idle disconnect forward. + if (!document.body.classList.contains("editing")) activityWs(); + clearTimeout(idleTimer); + idleTimer = setTimeout(() => { + if (ws) { + // Flush what is left unsent, then hang up. report() would try to + // reconnect a dead socket, which is exactly what we avoid here. + const left = takeReadTime(); + if (Math.round(left / 1000) > 0) ping({ read: left }); + ws.close(); + } + }, IDLE_MS); + // Frequently update the article read time on the server. + if (readElapsed + performance.now() - readStart >= READ_FLUSH_MS) { + ping({ read: takeReadTime() }); + } } function takeReadTime() { @@ -519,28 +587,19 @@ import "overlayscrollbars/overlayscrollbars.css"; clearTimeout(readInactivityTimer); } - function sendClosePing() { - if (closePingedFor === currentPath) return; - closePingedFor = currentPath; - const read = takeReadTime(); - if (Math.round(read / 1000) <= 0) return; - ping({ read, beacon: true }); - } - for (const ev of ["mousemove", "mousedown", "touchstart", "touchmove", "scroll", "keydown"]) { addEventListener(ev, markReadActivity, { passive: true }); } - addEventListener("pagehide", sendClosePing); - // The initial page load pings too — it is what starts the visit and + // The initial page load reports too — it is what starts the visit and // counts the entry page view (the document GET alone records nothing). // It carries only ``to``: the server attributes the entry to the referer // it saw on the document GET (unavailable to JS once loaded), and an // ``fr`` equal to ``to`` would log a bogus self-transition when a // session already exists (e.g. a second tab). // Sent once per load, after the auth probes so the admin gate applies; - // the pageshow re-probe must not ping again. Reloads are not visits: - // pinging them would double-count the view and log a self-transition. + // the pageshow re-probe must not report again. Reloads are not visits: + // reporting them would double-count the view and log a self-transition. let entryPinged = false; function pingEntryOnce() { if (entryPinged) return; @@ -770,7 +829,6 @@ import "overlayscrollbars/overlayscrollbars.css"; // External link: the browser navigates; record the full https URL so // different links to the same domain stay distinct in analytics. if (url.protocol === "https:") { - closePingedFor = currentPath; ping({ to: url.href, read: takeReadTime() }); } return; @@ -794,7 +852,6 @@ import "overlayscrollbars/overlayscrollbars.css"; const from = currentPath; load(url).then((ok) => { if (!ok) return; - closePingedFor = null; ping({ to: url.pathname, fr: from, read: takeReadTime() }); resetReadTime(); }); diff --git a/pagerite/analytics.py b/pagerite/analytics.py index 0231972..f70ac52 100644 --- a/pagerite/analytics.py +++ b/pagerite/analytics.py @@ -1,18 +1,23 @@ """Server-side visit analytics (collection only; see docs/analytics.md). -Events come from navigation pings POSTed to /_a by pagerite.js: the first -ping on page load starts a visit, later pings extend it, and pings with no -known session start a fresh one (missing data, not dropped). The document +Events come from pagerite.js over the /_ws WebSocket (``Ping`` messages as +JSON text frames): the first navigation message on page load starts a +visit, later messages extend it, and messages with no known session start a +fresh one (missing data, not dropped). Active reading time is reported as +frequent ``read`` updates while the user is active; the times are +cumulative per trail item, so a disconnect simply leaves the last logged +time in place. The document GET handler stashes the entry referer (external https origin) and any -utm_* query parameters in in-memory IP tables, consumed when the ping -starts the visit; nothing is counted without a ping (plain bots that only -fetch documents end up in the crawler list). JS-running crawlers -(Googlebot, GoogleOther, Applebot, ...) do ping, but their UA gives them -away (``_is_bot_ua``) and their pings are ignored, so they land in the +utm_* query parameters in in-memory IP tables, consumed when the first +message starts the visit; nothing is counted without a message (plain bots +that only fetch documents end up in the crawler list). JS-running crawlers +(Googlebot, GoogleOther, Applebot, ...) do connect and send messages, but +their UA gives them +away (``_is_bot_ua``) and their messages are ignored, so they land in the crawler list too. Idle-time link preloads from pagerite.js carry an -``x-pagerite-preload`` header and are not tracked at all — the ping sent -when the user actually navigates does the counting. -Admin clients ping with ``hide=1``: the client record is flagged ``hide``, +``x-pagerite-preload`` header and are not tracked at all — the navigation +message sent when the user actually navigates does the counting. +Admin clients send ``hide``: the client record is flagged ``hide``, which covers everything that client ever did — visits and crawler hits from before the login included. Aggregates (site visits, page views, transitions) are not stored; they are computed at display time from the @@ -70,6 +75,27 @@ def _compact_user_agent(ua: str) -> str: return " ".join(p for p in parts if p).strip() +class Ping(msgspec.Struct, omit_defaults=True): + """One client message on the /_ws activity WebSocket. + + Sent as a JSON text frame (msgspec-encoded, decoded to str for the + wire). ``to`` set: a navigation — internal page path or external https + exit URL. ``read`` alone (with ``fr``): an active reading-time update + for the page ``fr``; these arrive frequently while the user is active + and accumulate on the trail item. ``hide`` flags the client as an + admin: everything it ever did is excluded from the statistics. + """ + + #: Path of the page the activity happened on ("" for the initial load). + fr: str = "" + #: Navigation target: internal path or external https exit URL. + to: str = "" + #: Active reading time (seconds) spent on ``fr`` since the last report. + read: int = 0 + #: Admin client: record but hide everything from the statistics. + hide: bool = False + + class Client(msgspec.Struct, omit_defaults=True): """Client metadata shared by visits, crawler hits and abuse hits. @@ -151,7 +177,7 @@ class Visit(msgspec.Struct, omit_defaults=True): class CrawlerHit(msgspec.Struct, omit_defaults=True): - """A document GET that was never followed by an analytics ping. + """A document GET that was never followed by an activity message. Client metadata is held in ``Analytics.clients`` keyed by ``client``. """ @@ -392,19 +418,19 @@ class Store: #: client hash -> index of the current visit in data.visits self.sessions: dict[bytes, int] = {} #: ip -> external https origin of the latest document GET carrying - #: one, stashed for the visit the client's initial ping starts. + #: one, stashed for the visit the client's initial message starts. #: Internal or absent referers never touch the table. self.pending_referers: dict[str, str] = {} #: ip -> utm_* query parameters from the latest document GET that - #: carried any, stashed for the visit the client's initial ping starts. + #: carried any, stashed for the visit the client's initial message starts. #: Only non-empty sets are stored, so a later parameter-less page #: does not overwrite an earlier tagged landing URL. self.pending_utms: dict[str, dict[str, str]] = {} - #: Document GETs that have not yet been matched by a ping. Kept + #: Document GETs that have not yet been matched by a message. Kept #: in RAM only; expired entries are written to ``data.crawlers``. self.pending_crawlers: list[CrawlerHit] = [] #: client hash -> {path: status} for recent document GETs, consumed - #: by the matching ping to record the status of each visited path. + #: by the matching message to record the status of each visited path. self.pending_statuses: dict[bytes, dict[str, int]] = {} #: ip -> number of plain (non-telltale) 404s seen, in RAM only; #: reaching ``_ABUSE_404_THRESHOLD`` classifies the IP as abuse. @@ -721,16 +747,16 @@ class Store: ) -> list[bytes]: """Stash the entry referer/UTM tags and queue a pending crawler hit. - Nothing is counted here — the client's initial /_a ping starts the - visit (only non-admin clients ping). Only a cross-origin https + Nothing is counted here — the client's first /_ws message starts the + visit (only non-admin clients report). Only a cross-origin https referer updates the table; an internal or absent referer leaves any stashed origin untouched. UTM parameters are kept only when the landing URL actually carries them, so a subsequent parameter-less page does not erase an earlier tagged landing. - Every document GET is also queued as a pending crawler hit. If a ping - from the same client arrives within ``_CRAWLER_TIMEOUT``, the hit is - discarded; otherwise it is flushed to ``data.crawlers``. The + Every document GET is also queued as a pending crawler hit. If a + message from the same client arrives within ``_CRAWLER_TIMEOUT``, the + hit is discarded; otherwise it is flushed to ``data.crawlers``. The Accept-Language header is stored on the client record immediately; host/geoip are filled in later by async enrichment. @@ -794,15 +820,16 @@ class Store: hide: bool = False, read: int = 0, ) -> tuple[int | None, list[bytes]]: - """Record a client navigation ping ({from, to, read} from pagerite.js). + """Record a client activity message (``Ping`` from pagerite.js over /_ws). ``to`` is an internal path ("/...") or an https URL for exit links; a - missing/empty ``to`` means the page is being closed and only the + missing/empty ``to`` means a pure reading-time update and only the ``read`` time should be recorded. The transition is always counted when ``to`` is present; the trail only grows on first sight of a page within - the visit. ``read`` is the active time (seconds) spent on ``from_``. + the visit. ``read`` is the active time (seconds) spent on ``from_`` + since the previous report. - A ping with no known session starts a fresh visit, consuming the + A message with no known session starts a fresh visit, consuming the referer and UTM tags stashed by the document GET if there are any. ``hide`` is set by admin clients: the client record is flagged @@ -811,7 +838,7 @@ class Store: normally. Hidden clients are excluded from every statistic and list at display time, and their pending crawler hits are discarded. - Pings from IPs classified as abuse, and pings whose User-Agent + Messages from IPs classified as abuse, and messages whose User-Agent claims a JS-running crawler identity (``_is_bot_ua``), are ignored entirely — the crawler's pending hits stay queued and flush to ``data.crawlers`` normally. diff --git a/pagerite/app.py b/pagerite/app.py index 72e3473..0ef5d0b 100644 --- a/pagerite/app.py +++ b/pagerite/app.py @@ -32,10 +32,10 @@ from xml.sax.saxutils import escape as xml_escape import blake3 import httpx +import msgspec from fastapi import ( FastAPI, HTTPException, - Query, Request, WebSocket, WebSocketDisconnect, @@ -935,7 +935,7 @@ async def delete_page(path: str) -> None: _SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9_-]*$") -def _client_ip(request: Request) -> str: +def _client_ip(request: Request | WebSocket) -> str: """Client IP: first X-Forwarded-For hop (we sit behind a proxy), else the direct peer.""" forwarded = request.headers.get("x-forwarded-for", "").split(",")[0].strip() @@ -1108,48 +1108,54 @@ async def analytics_page(request: Request) -> Response: ) -@app.post("/_a", status_code=204) -async def analytics_ping( - request: Request, - fr: str = Query(""), - to: str | None = Query(None), - hide: int = Query(0), - read: int = Query(0), -) -> None: - """Record a navigation ping (?fr=&to=&hide=&read=); fire-and-forget. +@app.websocket("/_ws") +async def activity_ws(ws: WebSocket) -> None: + """Collect visitor activity: navigations and reading-time updates. - The initial page-load ping carries only ``to``: the entry is attributed - to the referer/UTM tags stashed by the document GET (see _track_entry), - which JS cannot see once the page has loaded. - - The reverse-DNS and DB-IP geoip lookups happen in a background task so - the response is never delayed by slow DNS or the first MMDB decompress. + Public, like the pages themselves (only /_api is gated); one connection + follows a browsing session. Messages are ``analytics.Ping`` structs as + JSON text frames; ``to`` set is a navigation, ``read`` alone a + reading-time update. The reverse-DNS and DB-IP geoip lookups happen in + background tasks so message handling is never delayed by slow DNS or + the first MMDB decompress. """ - ip = _client_ip(request) - visit_index, flushed_clients = analytics_store.ping( - fr, - to, - ip, - request.headers.get("user-agent", ""), - request.headers.get("accept-language", ""), - hide=bool(hide), - read=read, - ) - if visit_index is not None: - visit = analytics_store.data.visits[visit_index] - asyncio.create_task(_enrich_client(visit.client)) - _schedule_client_enrichment(flushed_clients) - _schedule_favicon_fetch() + await ws.accept() + ip = _client_ip(ws) + ua = ws.headers.get("user-agent", "") + accept_language = ws.headers.get("accept-language", "") + try: + while True: + text = await ws.receive_text() + try: + msg = msgspec.json.decode(text.encode(), type=analytics.Ping) + except msgspec.DecodeError: + continue + visit_index, flushed_clients = analytics_store.ping( + msg.fr, + msg.to or None, + ip, + ua, + accept_language, + hide=msg.hide, + read=msg.read, + ) + if visit_index is not None: + visit = analytics_store.data.visits[visit_index] + asyncio.create_task(_enrich_client(visit.client)) + _schedule_client_enrichment(flushed_clients) + _schedule_favicon_fetch() + except WebSocketDisconnect: + pass def _track_entry(path: str, request: Request, *, status: int = 200) -> list[bytes]: """Stash the referer/UTM tags and queue a pending crawler hit for the GET. - Nothing is counted on the GET itself — the client's /_a ping starts the - visit, so bots never register as visits (JS-running crawlers ping too, - but the ping handler ignores known bot UAs). (Admin clients ping too, - but with hide=1, which flags their visit hidden: it is recorded but - excluded from all statistics and from the crawler list.) + Nothing is counted on the GET itself — the client's first /_ws message + starts the visit, so bots never register as visits (JS-running crawlers + connect too, but the WebSocket handler ignores known bot UAs). (Admin + clients report too, but with hide, which flags their visit hidden: it is + recorded but excluded from all statistics and from the crawler list.) The devserver's health probe (``GET /?from=devserver.py`` from ``127.0.0.1``) is ignored: it is not real traffic and would otherwise be @@ -1161,7 +1167,8 @@ def _track_entry(path: str, request: Request, *, status: int = 200) -> list[byte """ if request.headers.get("x-pagerite-preload"): # Idle-time page-cache warm-up by pagerite.js, not a page view: the - # ping sent when the user actually navigates does the counting. + # activity message sent when the user actually navigates does the + # counting. # (Forging the header only hides a GET from the crawler stats; the # path-based abuse classification is unaffected.) return []