Collect analytics over a /_ws WebSocket instead of POST /_a pings

docs/analytics.md

@@ -9,9 +9,9 @@ directory, e.g. `localhost/analytics.json`).
   `CrawlerHit`, `AbuseHit`, `Favicon`) and the `Store` (in-memory data + session map,
   atomic JSON persistence).
 - `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`),
-  the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics`
+  the `/_ws` activity WebSocket, and `WebSocket /_api/ws/analytics`
   (admin-gated like every `/_api` endpoint).
-- `frontend/src/pagerite.js` — client navigation pings and the 📊 pen.
+- `frontend/src/pagerite.js` — the client activity channel and the 📊 pen.
 - `frontend/src/AnalyticsView.vue` — viewer component rendered inside the
   normal site layout on the `/_a` analytics page.
 - `frontend/src/analytics-main.js` — page entry that mounts `AnalyticsView`
@@ -19,46 +19,61 @@ directory, e.g. `localhost/analytics.json`).
 
 ## What is collected
 
-The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
+The client (`pagerite.js`) keeps a WebSocket connection to `/_ws` for the
-`fr`, `to`, `hide` and `read` as query parameters (`fr` = source path;
+whole browsing session and sends activity messages over it — JSON text
-falsy values are omitted):
+frames matching the server's `Ping` msgspec struct with the fields `fr`
+(source path), `to` (navigation target), `read` (active seconds on `fr`
+since the last report) and `hide`; falsy fields are omitted. One channel
+follows the session, so the activity of a visit stays tied together, and
+while the user is active the accumulated reading time is flushed every few
+seconds: the trail times are cumulative, so a disconnection simply leaves
+the last reported time in place (no close beacon). After 5 minutes without
+any activity the client closes the socket itself — a sleeping browser tab
+would lose it anyway — and the next activity reconnects as a fresh session;
+reconnects are attempted only on user activity, with an exponential backoff
+between attempts so a failing endpoint is never hammered. Idle-time link preloads
+stay plain `fetch()` calls so the browser may cache the responses; the
+WebSocket reports actual navigations and active time spent on a page.
 
 - **Initial page load**: only `to` — the loaded path — is sent, never `fr`
   (an `fr` equal to `to` would log a bogus self-transition when a session
-  already exists, e.g. a second tab). This ping is what starts
+  already exists, e.g. a second tab). This message is what starts
   the visit and counts the entry page view — the document GET alone records
   nothing, so bots never register (admin browsing does register, but
   flagged `hide`; see **Admins** below). JS-running crawlers
-  (Googlebot, GoogleOther, Applebot, ...) do ping, but their User-Agent
+  (Googlebot, GoogleOther, Applebot, ...) do connect and report, but their
-  gives them away: pings whose UA matches `_is_bot_ua` (anything calling
+  User-Agent gives them away: messages whose UA matches `_is_bot_ua`
+  (anything calling
   itself a "bot", plus known exceptions such as GoogleOther) are ignored
   server-side, and their document GETs land in the crawler list instead.
   No source-IP verification is done: a spoofed bot UA merely lands in the
   crawler stats, and scanners that probe telltale paths are caught by the
   abuse rules regardless. Reloads are not
-  visits: the ping is skipped (PerformanceNavigationTiming `reload`), so a
+  visits: the message is skipped (PerformanceNavigationTiming `reload`), so a
   refresh neither counts a second view nor logs a self-transition. The GET
   handler stashes a cross-origin https `Referer` (origin part only —
   unavailable to JS once the page has loaded) and any
-  `utm_*` query parameters in in-memory IP tables, consumed by the ping that
+  `utm_*` query parameters in in-memory IP tables, consumed by the first
+  message that
   starts the visit; internal or absent referers never touch the referer table.
 - **Internal fetch-navigations**: `to` is the target path, sent only after
   the swap actually happened (a failed swap falls back to a full load,
-  whose initial ping counts the view instead — no gap, no double count).
+  whose initial message counts the view instead — no gap, no double count).
 - **External links** (`https` only): `to` is the link's full URL. This is the
   exit-link record; the user may continue navigating afterwards (new tab,
   back), so the exit URL is not necessarily the last trail entry. Outbound
   links are stored by full URL so several links to the same domain remain
   distinct.
 - **Excluded**: back/forward (popstate) navigations, navigating *to* the
-  analytics page (`/_a` — its GET is untracked, and the server rejects it
+  analytics page (`/_a` — its GET is untracked, and the server cannot
-  as a ping target anyway), and everything while the user has the editor
+  record it as a navigation target anyway), and everything while the user has
+  the editor
   open (`body.editing`). Admin noise, not visits. Navigating *away* from
-  `/_a` does ping: the fetch-navigation already GET-ed the target page
+  `/_a` does report: the fetch-navigation already GET-ed the target page
-  without the preload header, and without the ping that GET would flush to
+  without the preload header, and without the message that GET would flush to
   the crawler list.
 - **Admins**: when SSO is in use and the session is known to be an admin,
-  the client still pings but adds `hide=1`. The activity is recorded as
+  the client still reports but adds `hide`. The activity is recorded as
   usual (navigations and all), but the `hide` flag is set on the **client
   record** — so it covers everything that client ever did: visits and
   crawler hits from before the login included. Hidden clients never appear
@@ -81,7 +96,7 @@ falsy values are omitted):
   extension matching the actual MIME).  The origin → file name mapping is
   recorded in `Analytics.favicons` (`Favicon.file`/`fetched`); misses are
   recorded too and retried only after 7 days.  Fetches are scheduled after
-  each ping and once at startup, which backfills icons for already-recorded
+  each activity message and once at startup, which backfills icons for already-recorded
   data.  The viewer payload carries `favicons` (origin → `/_f/...` path),
   and the viewer shows the icon wherever an external site is mentioned:
   referer/exit trail links in the visit table and the source/exit pills of
@@ -100,8 +115,8 @@ falsy values are omitted):
   `host`; local/reserved/multicast addresses are skipped.  If a DB-IP MMDB
   file (`dbip-*.mmdb` or `dbip-*.mmdb.gz`) is present in the repository
   root, it is loaded at startup and used to look up `country`/`city`.  These
-  lookups run in background tasks after the event is stored, so the `/_a`
+  lookups run in background tasks after the event is stored, so WebSocket
-  response is never delayed.  The decompressed `dbip-*.mmdb` file is kept in
+  message handling is never delayed.  The decompressed `dbip-*.mmdb` file is kept in
   the repository root and ignored by git.  The CLI flag `--dbip`
   (`uv run pagerite --dbip`) downloads the latest
   `dbip-city-lite-YYYY-MM.mmdb.gz` from DB-IP before the server starts,
@@ -110,10 +125,11 @@ falsy values are omitted):
   file is used.
 - **Crawler hits**: every document GET is queued in RAM as a pending crawler
   hit — except idle-time link preloads from pagerite.js, which carry an
-  `x-pagerite-preload` header and are not tracked at all (the ping sent when
+  `x-pagerite-preload` header and are not tracked at all (the navigation
-  the user actually navigates to a preloaded page does the counting; forging
+  message sent when the user actually navigates to a preloaded page does
+  the counting; forging
   the header only hides a GET from the crawler stats, the path-based abuse
-  classification is unaffected).  If a ping
+  classification is unaffected).  If a message
   from the same client arrives within 10 seconds the hit is discarded;
   otherwise it is written to `crawlers` — unless the client is hidden
   (admin), in which case the hit is discarded on expiry too.  Crawlers do not count as
@@ -131,7 +147,7 @@ falsy values are omitted):
   random-UA scanner no longer pollutes the crawler stats of the legitimate
   bot it impersonates.  Once classified, every document GET and 404 from the
   IP is recorded as an abuse hit with the full request path (query string
-  included), and its pings are ignored.  The classified IP set (`abuse_ips`)
+  included), and its activity messages are ignored.  The classified IP set (`abuse_ips`)
   is persisted in the JSON file; the plain-404 counters are RAM-only.  In the
   viewer, abuse hits are grouped by IP (never by client/UA — scanners
   randomize theirs) in a separate "Abuse" table.  Identical paths are
@@ -145,9 +161,9 @@ falsy values are omitted):
 There are no cookies. A visit is tied together by a client hash — the first
 6 bytes of a blake3 digest over the prettified IP (IPv4 unchanged, IPv6
 /64 network), the raw `User-Agent` string and the extracted
-`Accept-Language` tag.  The first ping from a client hash starts a new
+`Accept-Language` tag.  The first message from a client hash starts a new
-visit; subsequent pings extend it.  Pings arriving with no known session
+visit; subsequent messages extend it.  Messages arriving with no known session
-(server restart) start a fresh visit from the first ping — treated as
+(server restart) start a fresh visit from the first message — treated as
 missing data rather than dropped.  The client-hash → visit map and the IP →
 entry-referer/UTM tables are in-memory only; client metadata is stored in
 `Analytics.clients` keyed by the client hash.
@@ -164,7 +180,7 @@ Each `Client` record:
 - `ua` — raw `User-Agent` string,
 - `ua_pretty` — compact display form of the UA (browser/OS/device) when
   parsable, otherwise the raw string,
-- `hide` — true for admin clients (`hide=1` ping): all their visits,
+- `hide` — true for admin clients (`hide` message field): all their visits,
   crawler hits and abuse hits are recorded but excluded from every
   statistic and from the viewer payload.
 
@@ -180,7 +196,7 @@ Each `Visit` record:
   reading time in seconds (`read`) and the most recent HTTP status seen
   for the target (`status`). Re-visiting an already seen target updates
   its item instead of appending.
-- `navs` — every navigation ping (`fr`, `to`), keyed by its timestamp,
+- `navs` — every navigation message (`fr`, `to`), keyed by its timestamp,
   repeats included. The aggregates are computed from this log at display
   time.
 - `utm` — `utm_*` query parameters from the landing URL, as a dict.
@@ -227,7 +243,8 @@ shapes, part of the WebSocket payload (`Display` struct alongside `visits`,
 
 - `transitions`: time series of page transitions, sparse nested dict
   `from -> to -> bucket -> count` with 5-minute bucketing. `from` is the
-  referer origin or `"(direct)"` for initial loads, a page path for pings.
+  referer origin or `"(direct)"` for initial loads, a page path for
+  navigations.
 - `views`: time series of page loads, `path -> bucket -> count`, sparse: only
   non-zero 5-minute buckets exist (bucket key is its floored ISO timestamp).
   Every load counts, including repeats within a visit; external exit origins

frontend/src/pagerite.js

@@ -347,8 +347,8 @@ import "overlayscrollbars/overlayscrollbars.css";
     for (const url of urls) {
       if (pageCache.has(url)) continue;
       // x-pagerite-preload: idle cache warm-up, not a page view — the
-      // server excludes these GETs from analytics (the ping sent on actual
+      // server excludes these GETs from analytics (the navigation message
-      // navigation does the counting).
+      // sent on actual navigation does the counting).
       fetch(url, { headers: { "x-pagerite-preload": "1" } })
         .then((r) => (r.ok && (r.headers.get("content-type") || "").includes("text/html")
           ? r.text() : ""))
@@ -432,61 +432,111 @@ import "overlayscrollbars/overlayscrollbars.css";
     });
   }, { passive: true });
 
-  // --- Analytics pings ---------------------------------------------------
+  // --- Analytics over WebSocket ------------------------------------------
-  // Fire-and-forget POSTs to /_a with the fields as query parameters (a
+  // One /_ws connection follows the whole browsing session: the initial page
-  // beacon can carry no body, and query args show in server logs next to
+  // load (starts the visit — the server counts nothing from the document GET
-  // the document GET they refer to): on the initial page load (starts the
+  // alone), internal fetch-navigations, external https exits, and frequent
-  // visit — the server counts nothing from the document GET alone), for
+  // active reading-time updates. Messages are JSON text frames matching the
-  // internal fetch-navigations, for external https exits, and on window
+  // server's msgspec Ping struct: {fr?, to?, read?, hide?} — falsy fields
-  // close. ``read`` is the active time (ms) spent on ``fr``.
+  // are omitted. ``read`` is the active time (ms) accumulated on ``fr`` since
-  // Reading time pauses after 1 minute of inactivity and resumes on the
+  // the last report; reading time pauses after 1 minute of inactivity and
-  // next mouse/touch/scroll/keyboard event.
+  // resumes on the next mouse/touch/scroll/keyboard event. While the user is
-  // Excluded: back/forward (popstate never pings), everything while the
+  // active, accumulated reading time is flushed every few seconds, so a
+  // disconnect simply leaves the last reported time on the server — no close
+  // beacon is needed. After 5 minutes without any activity the client closes
+  // the channel itself (a sleeping tab would lose it anyway); the next
+  // activity reconnects and the server sees a new session.
+  // Excluded: back/forward (popstate never reports), everything while the
   // editor is open (body.editing — admin noise, not visits), and
-  // navigations TO the analytics page (/_a — admin machinery, and the
+  // navigations TO the analytics page (/_a — admin machinery). Navigations
-  // server rejects it as a ping target anyway). Navigations AWAY from /_a
+  // AWAY from /_a must report: load() already fetched the target page
-  // must ping: load() already fetched the target page without the preload
+  // without the preload header, and without the message that GET would flush
-  // header, and without the ping that GET would flush to the crawler list.
+  // to the crawler list.
   // Admins (when SSO is actually in use — with no auth proxy "admin" is
-  // everyone's state) ping normally but with hide=1: the server then
+  // everyone's state) report normally but with hide: the server then flags
-  // records nothing and scrubs any session the same browser accumulated
+  // the client record, scrubbing everything it ever did from the statistics,
-  // before logging in, so admins never show up as visits or crawlers.
+  // so admins never show up as visits or crawlers.
   // See docs/analytics.md.
 
-  // fetch wrapper: every key of ``params`` becomes a query arg on /_a
+  // The activity WebSocket. Messages sent before the connection opens are
-  // (falsy values are omitted). Admins get hide=1. ``beacon`` uses
+  // queued (the queue keeps the interim activity). Reconnects are driven by
-  // sendBeacon when available, for unload-time pings.
+  // user activity only — never by timers while the page sits idle — with an
-  function pingFetch(params, { beacon = false } = {}) {
+  // exponential falloff between attempts so a failing endpoint cannot make
-    const query = new URLSearchParams();
+  // us hammer the server (or trip its security limits). After a longer
-    if (ssoAvailable && isAdmin) params = { ...params, hide: 1 };
+  // stretch without any activity we close the socket proactively: the user
-    for (const [key, value] of Object.entries(params)) {
+  // has moved on and left the tab open (a sleeping browser tab would lose
-      if (value) query.set(key, value);
+  // the connection anyway), so the next activity reconnects and registers
-    }
+  // as a fresh session. Analytics must never break navigation: every send
-    const url = `/_a?${query}`;
+  // is wrapped, and a server without the endpoint just leaves the socket
+  // failing in the background.
+  let ws = null;
+  const wsQueue = [];
+  let wsReconnectMs = 1000;
+  let wsNotBefore = 0;
+
+  function activityWs() {
+    if (ws || Date.now() < wsNotBefore) return;
+    const url = new URL("/_ws", location.href);
+    url.protocol = url.protocol === "https:" ? "wss:" : "ws:";
     try {
-      if (beacon && navigator.sendBeacon) {
+      ws = new WebSocket(url);
-        navigator.sendBeacon(url);
+    } catch {
-      } else {
+      return;
-        fetch(url, { method: "POST", keepalive: true });
+    }
-      }
+    ws.onopen = () => {
-    } catch { /* analytics must never break navigation */ }
+      wsReconnectMs = 1000;
+      for (const msg of wsQueue.splice(0)) ws.send(JSON.stringify(msg));
+    };
+    ws.onclose = () => {
+      ws = null;
+      // No timer here: the next user activity retries, after the backoff.
+      wsNotBefore = Date.now() + wsReconnectMs;
+      wsReconnectMs = Math.min(wsReconnectMs * 2, 30_000);
+    };
+    ws.onerror = () => ws.close();
   }
 
-  function ping({ to, fr = currentPath, read = 0, beacon = false } = {}) {
+  function report(msg) {
     if (document.body.classList.contains("editing")) return;
-    if (to === "/_a") return;
+    if (msg.to === "/_a") return;
-    pingFetch({ fr, to, read: Math.round(read / 1000) }, { beacon });
+    if (ssoAvailable && isAdmin) msg.hide = true;
+    activityWs();
+    if (ws?.readyState === WebSocket.OPEN) {
+      try {
+        ws.send(JSON.stringify(msg));
+        return;
+      } catch { /* fall through to queueing */ }
+    }
+    wsQueue.push(msg);
+  }
+
+  function ping({ to, fr = currentPath, read = 0 } = {}) {
+    // Reading-time updates from the analytics page itself are not tracked
+    // (/_a is admin machinery; the server would reject the path anyway).
+    if (!to && currentPath === "/_a") return;
+    const msg = {};
+    if (fr) msg.fr = fr;
+    if (to) msg.to = to;
+    const secs = Math.round(read / 1000);
+    if (secs > 0) msg.read = secs;
+    if (!msg.to && !msg.read) return;
+    report(msg);
   }
 
   // Active reading time for the current page. The clock stops after 1 minute
   // without activity and restarts on the next mouse/touch/scroll/keyboard
-  // event.
+  // event. Every READ_FLUSH_MS of accumulated activity is reported. After
+  // IDLE_MS with no activity at all, the remaining read time is flushed and
+  // the WebSocket is closed: the user has moved on, and the next activity
+  // reconnects as a new session.
   const INACTIVE_MS = 60_000;
+  const READ_FLUSH_MS = 5_000;
+  const IDLE_MS = 5 * 60_000;
   let readStart = performance.now();
   let readElapsed = 0;
   let reading = true;
   let readInactivityTimer = null;
-  let closePingedFor = null;
+  let idleTimer = null;
 
   function markReadActivity() {
     if (!reading) {
@@ -500,6 +550,24 @@ import "overlayscrollbars/overlayscrollbars.css";
         reading = false;
       }
     }, INACTIVE_MS);
+    // Any activity is a sign of life: (re)connect the channel if it was
+    // dropped or idle-closed (not while editing — admin noise), and push
+    // the idle disconnect forward.
+    if (!document.body.classList.contains("editing")) activityWs();
+    clearTimeout(idleTimer);
+    idleTimer = setTimeout(() => {
+      if (ws) {
+        // Flush what is left unsent, then hang up. report() would try to
+        // reconnect a dead socket, which is exactly what we avoid here.
+        const left = takeReadTime();
+        if (Math.round(left / 1000) > 0) ping({ read: left });
+        ws.close();
+      }
+    }, IDLE_MS);
+    // Frequently update the article read time on the server.
+    if (readElapsed + performance.now() - readStart >= READ_FLUSH_MS) {
+      ping({ read: takeReadTime() });
+    }
   }
 
   function takeReadTime() {
@@ -519,28 +587,19 @@ import "overlayscrollbars/overlayscrollbars.css";
     clearTimeout(readInactivityTimer);
   }
 
-  function sendClosePing() {
-    if (closePingedFor === currentPath) return;
-    closePingedFor = currentPath;
-    const read = takeReadTime();
-    if (Math.round(read / 1000) <= 0) return;
-    ping({ read, beacon: true });
-  }
-
   for (const ev of ["mousemove", "mousedown", "touchstart", "touchmove", "scroll", "keydown"]) {
     addEventListener(ev, markReadActivity, { passive: true });
   }
-  addEventListener("pagehide", sendClosePing);
 
-  // The initial page load pings too — it is what starts the visit and
+  // The initial page load reports too — it is what starts the visit and
   // counts the entry page view (the document GET alone records nothing).
   // It carries only ``to``: the server attributes the entry to the referer
   // it saw on the document GET (unavailable to JS once loaded), and an
   // ``fr`` equal to ``to`` would log a bogus self-transition when a
   // session already exists (e.g. a second tab).
   // Sent once per load, after the auth probes so the admin gate applies;
-  // the pageshow re-probe must not ping again. Reloads are not visits:
+  // the pageshow re-probe must not report again. Reloads are not visits:
-  // pinging them would double-count the view and log a self-transition.
+  // reporting them would double-count the view and log a self-transition.
   let entryPinged = false;
   function pingEntryOnce() {
     if (entryPinged) return;
@@ -770,7 +829,6 @@ import "overlayscrollbars/overlayscrollbars.css";
       // External link: the browser navigates; record the full https URL so
       // different links to the same domain stay distinct in analytics.
       if (url.protocol === "https:") {
-        closePingedFor = currentPath;
         ping({ to: url.href, read: takeReadTime() });
       }
       return;
@@ -794,7 +852,6 @@ import "overlayscrollbars/overlayscrollbars.css";
     const from = currentPath;
     load(url).then((ok) => {
       if (!ok) return;
-      closePingedFor = null;
       ping({ to: url.pathname, fr: from, read: takeReadTime() });
       resetReadTime();
     });

pagerite/analytics.py

@@ -1,18 +1,23 @@
 """Server-side visit analytics (collection only; see docs/analytics.md).
 
-Events come from navigation pings POSTed to /_a by pagerite.js: the first
+Events come from pagerite.js over the /_ws WebSocket (``Ping`` messages as
-ping on page load starts a visit, later pings extend it, and pings with no
+JSON text frames): the first navigation message on page load starts a
-known session start a fresh one (missing data, not dropped). The document
+visit, later messages extend it, and messages with no known session start a
+fresh one (missing data, not dropped). Active reading time is reported as
+frequent ``read`` updates while the user is active; the times are
+cumulative per trail item, so a disconnect simply leaves the last logged
+time in place. The document
 GET handler stashes the entry referer (external https origin) and any
-utm_* query parameters in in-memory IP tables, consumed when the ping
+utm_* query parameters in in-memory IP tables, consumed when the first
-starts the visit; nothing is counted without a ping (plain bots that only
+message starts the visit; nothing is counted without a message (plain bots
-fetch documents end up in the crawler list).  JS-running crawlers
+that only fetch documents end up in the crawler list).  JS-running crawlers
-(Googlebot, GoogleOther, Applebot, ...) do ping, but their UA gives them
+(Googlebot, GoogleOther, Applebot, ...) do connect and send messages, but
-away (``_is_bot_ua``) and their pings are ignored, so they land in the
+their UA gives them
+away (``_is_bot_ua``) and their messages are ignored, so they land in the
 crawler list too.  Idle-time link preloads from pagerite.js carry an
-``x-pagerite-preload`` header and are not tracked at all — the ping sent
+``x-pagerite-preload`` header and are not tracked at all — the navigation
-when the user actually navigates does the counting.
+message sent when the user actually navigates does the counting.
-Admin clients ping with ``hide=1``: the client record is flagged ``hide``,
+Admin clients send ``hide``: the client record is flagged ``hide``,
 which covers everything that client ever did — visits and crawler hits
 from before the login included.  Aggregates (site visits, page views,
 transitions) are not stored; they are computed at display time from the
@@ -70,6 +75,27 @@ def _compact_user_agent(ua: str) -> str:
     return " ".join(p for p in parts if p).strip()
 
 
+class Ping(msgspec.Struct, omit_defaults=True):
+    """One client message on the /_ws activity WebSocket.
+
+    Sent as a JSON text frame (msgspec-encoded, decoded to str for the
+    wire).  ``to`` set: a navigation — internal page path or external https
+    exit URL.  ``read`` alone (with ``fr``): an active reading-time update
+    for the page ``fr``; these arrive frequently while the user is active
+    and accumulate on the trail item.  ``hide`` flags the client as an
+    admin: everything it ever did is excluded from the statistics.
+    """
+
+    #: Path of the page the activity happened on ("" for the initial load).
+    fr: str = ""
+    #: Navigation target: internal path or external https exit URL.
+    to: str = ""
+    #: Active reading time (seconds) spent on ``fr`` since the last report.
+    read: int = 0
+    #: Admin client: record but hide everything from the statistics.
+    hide: bool = False
+
+
 class Client(msgspec.Struct, omit_defaults=True):
     """Client metadata shared by visits, crawler hits and abuse hits.
 
@@ -151,7 +177,7 @@ class Visit(msgspec.Struct, omit_defaults=True):
 
 
 class CrawlerHit(msgspec.Struct, omit_defaults=True):
-    """A document GET that was never followed by an analytics ping.
+    """A document GET that was never followed by an activity message.
 
     Client metadata is held in ``Analytics.clients`` keyed by ``client``.
     """
@@ -392,19 +418,19 @@ class Store:
         #: client hash -> index of the current visit in data.visits
         self.sessions: dict[bytes, int] = {}
         #: ip -> external https origin of the latest document GET carrying
-        #: one, stashed for the visit the client's initial ping starts.
+        #: one, stashed for the visit the client's initial message starts.
         #: Internal or absent referers never touch the table.
         self.pending_referers: dict[str, str] = {}
         #: ip -> utm_* query parameters from the latest document GET that
-        #: carried any, stashed for the visit the client's initial ping starts.
+        #: carried any, stashed for the visit the client's initial message starts.
         #: Only non-empty sets are stored, so a later parameter-less page
         #: does not overwrite an earlier tagged landing URL.
         self.pending_utms: dict[str, dict[str, str]] = {}
-        #: Document GETs that have not yet been matched by a ping.  Kept
+        #: Document GETs that have not yet been matched by a message.  Kept
         #: in RAM only; expired entries are written to ``data.crawlers``.
         self.pending_crawlers: list[CrawlerHit] = []
         #: client hash -> {path: status} for recent document GETs, consumed
-        #: by the matching ping to record the status of each visited path.
+        #: by the matching message to record the status of each visited path.
         self.pending_statuses: dict[bytes, dict[str, int]] = {}
         #: ip -> number of plain (non-telltale) 404s seen, in RAM only;
         #: reaching ``_ABUSE_404_THRESHOLD`` classifies the IP as abuse.
@@ -721,16 +747,16 @@ class Store:
     ) -> list[bytes]:
         """Stash the entry referer/UTM tags and queue a pending crawler hit.
 
-        Nothing is counted here — the client's initial /_a ping starts the
+        Nothing is counted here — the client's first /_ws message starts the
-        visit (only non-admin clients ping). Only a cross-origin https
+        visit (only non-admin clients report). Only a cross-origin https
         referer updates the table; an internal or absent referer leaves any
         stashed origin untouched.  UTM parameters are kept only when the
         landing URL actually carries them, so a subsequent parameter-less page
         does not erase an earlier tagged landing.
 
-        Every document GET is also queued as a pending crawler hit.  If a ping
+        Every document GET is also queued as a pending crawler hit.  If a
-        from the same client arrives within ``_CRAWLER_TIMEOUT``, the hit is
+        message from the same client arrives within ``_CRAWLER_TIMEOUT``, the
-        discarded; otherwise it is flushed to ``data.crawlers``.  The
+        hit is discarded; otherwise it is flushed to ``data.crawlers``.  The
         Accept-Language header is stored on the client record immediately;
         host/geoip are filled in later by async enrichment.
 
@@ -794,15 +820,16 @@ class Store:
         hide: bool = False,
         read: int = 0,
     ) -> tuple[int | None, list[bytes]]:
-        """Record a client navigation ping ({from, to, read} from pagerite.js).
+        """Record a client activity message (``Ping`` from pagerite.js over /_ws).
 
         ``to`` is an internal path ("/...") or an https URL for exit links; a
-        missing/empty ``to`` means the page is being closed and only the
+        missing/empty ``to`` means a pure reading-time update and only the
         ``read`` time should be recorded. The transition is always counted when
         ``to`` is present; the trail only grows on first sight of a page within
-        the visit. ``read`` is the active time (seconds) spent on ``from_``.
+        the visit. ``read`` is the active time (seconds) spent on ``from_``
+        since the previous report.
 
-        A ping with no known session starts a fresh visit, consuming the
+        A message with no known session starts a fresh visit, consuming the
         referer and UTM tags stashed by the document GET if there are any.
 
         ``hide`` is set by admin clients: the client record is flagged
@@ -811,7 +838,7 @@ class Store:
         normally.  Hidden clients are excluded from every statistic and list
         at display time, and their pending crawler hits are discarded.
 
-        Pings from IPs classified as abuse, and pings whose User-Agent
+        Messages from IPs classified as abuse, and messages whose User-Agent
         claims a JS-running crawler identity (``_is_bot_ua``), are ignored
         entirely — the crawler's pending hits stay queued and flush to
         ``data.crawlers`` normally.

pagerite/app.py

@@ -32,10 +32,10 @@ from xml.sax.saxutils import escape as xml_escape
 
 import blake3
 import httpx
+import msgspec
 from fastapi import (
     FastAPI,
     HTTPException,
-    Query,
     Request,
     WebSocket,
     WebSocketDisconnect,
@@ -935,7 +935,7 @@ async def delete_page(path: str) -> None:
 _SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9_-]*$")
 
 
-def _client_ip(request: Request) -> str:
+def _client_ip(request: Request | WebSocket) -> str:
     """Client IP: first X-Forwarded-For hop (we sit behind a proxy), else
     the direct peer."""
     forwarded = request.headers.get("x-forwarded-for", "").split(",")[0].strip()
@@ -1108,48 +1108,54 @@ async def analytics_page(request: Request) -> Response:
     )
 
 
-@app.post("/_a", status_code=204)
+@app.websocket("/_ws")
-async def analytics_ping(
+async def activity_ws(ws: WebSocket) -> None:
-    request: Request,
+    """Collect visitor activity: navigations and reading-time updates.
-    fr: str = Query(""),
-    to: str | None = Query(None),
-    hide: int = Query(0),
-    read: int = Query(0),
-) -> None:
-    """Record a navigation ping (?fr=&to=&hide=&read=); fire-and-forget.
 
-    The initial page-load ping carries only ``to``: the entry is attributed
+    Public, like the pages themselves (only /_api is gated); one connection
-    to the referer/UTM tags stashed by the document GET (see _track_entry),
+    follows a browsing session.  Messages are ``analytics.Ping`` structs as
-    which JS cannot see once the page has loaded.
+    JSON text frames; ``to`` set is a navigation, ``read`` alone a
-
+    reading-time update.  The reverse-DNS and DB-IP geoip lookups happen in
-    The reverse-DNS and DB-IP geoip lookups happen in a background task so
+    background tasks so message handling is never delayed by slow DNS or
-    the response is never delayed by slow DNS or the first MMDB decompress.
+    the first MMDB decompress.
     """
-    ip = _client_ip(request)
+    await ws.accept()
-    visit_index, flushed_clients = analytics_store.ping(
+    ip = _client_ip(ws)
-        fr,
+    ua = ws.headers.get("user-agent", "")
-        to,
+    accept_language = ws.headers.get("accept-language", "")
-        ip,
+    try:
-        request.headers.get("user-agent", ""),
+        while True:
-        request.headers.get("accept-language", ""),
+            text = await ws.receive_text()
-        hide=bool(hide),
+            try:
-        read=read,
+                msg = msgspec.json.decode(text.encode(), type=analytics.Ping)
-    )
+            except msgspec.DecodeError:
-    if visit_index is not None:
+                continue
-        visit = analytics_store.data.visits[visit_index]
+            visit_index, flushed_clients = analytics_store.ping(
-        asyncio.create_task(_enrich_client(visit.client))
+                msg.fr,
-    _schedule_client_enrichment(flushed_clients)
+                msg.to or None,
-    _schedule_favicon_fetch()
+                ip,
+                ua,
+                accept_language,
+                hide=msg.hide,
+                read=msg.read,
+            )
+            if visit_index is not None:
+                visit = analytics_store.data.visits[visit_index]
+                asyncio.create_task(_enrich_client(visit.client))
+            _schedule_client_enrichment(flushed_clients)
+            _schedule_favicon_fetch()
+    except WebSocketDisconnect:
+        pass
 
 
 def _track_entry(path: str, request: Request, *, status: int = 200) -> list[bytes]:
     """Stash the referer/UTM tags and queue a pending crawler hit for the GET.
 
-    Nothing is counted on the GET itself — the client's /_a ping starts the
+    Nothing is counted on the GET itself — the client's first /_ws message
-    visit, so bots never register as visits (JS-running crawlers ping too,
+    starts the visit, so bots never register as visits (JS-running crawlers
-    but the ping handler ignores known bot UAs).  (Admin clients ping too,
+    connect too, but the WebSocket handler ignores known bot UAs).  (Admin
-    but with hide=1, which flags their visit hidden: it is recorded but
+    clients report too, but with hide, which flags their visit hidden: it is
-    excluded from all statistics and from the crawler list.)
+    recorded but excluded from all statistics and from the crawler list.)
 
     The devserver's health probe (``GET /?from=devserver.py`` from
     ``127.0.0.1``) is ignored: it is not real traffic and would otherwise be
@@ -1161,7 +1167,8 @@ def _track_entry(path: str, request: Request, *, status: int = 200) -> list[byte
     """
     if request.headers.get("x-pagerite-preload"):
         # Idle-time page-cache warm-up by pagerite.js, not a page view: the
-        # ping sent when the user actually navigates does the counting.
+        # activity message sent when the user actually navigates does the
+        # counting.
         # (Forging the header only hides a GET from the crawler stats; the
         # path-based abuse classification is unaffected.)
         return []