Fix crawler misclassification from /_a and orphan counts on visit scrub

Two analytics corrections verified against the production capture:

- pagerite.js suppressed pings with fr == '/_a', but fetch-navigation
  away from the analytics page had already GET-ed the target without the
  preload header; the orphaned pending hit then flushed to the crawler
  list, classifying a real user as a crawler. Navigations away from /_a
  now ping normally (the server rejects /_a as a target regardless, and
  admin noise is already handled by hide=1).

- _remove_visit only reversed the visit's creation counts, leaving
  views/transitions from later pings behind as orphans on the graph with
  no matching row in the visitor table. An in-memory per-visit count log
  now tracks every count event, so an admin hide=1 scrub reverses the
  visit completely.

docs/analytics.md

@@ -45,16 +45,19 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
   back), so the exit URL is not necessarily the last trail entry. Outbound
   links are stored by full URL so several links to the same domain remain
   distinct.
-- **Excluded**: back/forward (popstate) navigations, navigation involving
-  the analytics page itself (`/_a`), and everything while the user has the
-  editor open (`body.editing`). Admin noise, not visits.
+- **Excluded**: back/forward (popstate) navigations, navigating *to* the
+  analytics page (`/_a` — its GET is untracked, and the server rejects it
+  as a ping target anyway), and everything while the user has the editor
+  open (`body.editing`). Admin noise, not visits. Navigating *away* from
+  `/_a` does ping: the fetch-navigation already GET-ed the target page
+  without the preload header, and without the ping that GET would flush to
+  the crawler list.
 - **Admins**: when SSO is in use and the session is known to be an admin,
   the client still pings but adds `hide=1`. The server then records
   nothing — and if the same client session already had a visit from before
-  logging in, that visit is removed from the JSON along with the counts
-  recorded when it was created (site visit, entry view, entry transition).
-  Views/transitions logged by later pings inside such a visit lack
-  per-event timestamps and are left as-is. With no auth proxy (dev/test)
+  logging in, that visit is removed from the JSON along with every count
+  it recorded — an in-memory per-visit log of count events makes full
+  reversal possible. With no auth proxy (dev/test)
   "admin" is everyone's state, so `hide` stays 0 and everything is recorded.
 - The server validates `to`: internal paths must be valid slug paths
   ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the

frontend/src/pagerite.js

@@ -385,9 +385,11 @@ import "overlayscrollbars/overlayscrollbars.css";
   // Reading time pauses after 1 minute of inactivity and resumes on the
   // next mouse/touch/scroll/keyboard event.
   // Excluded: back/forward (popstate never pings), everything while the
-  // editor is open (body.editing — admin noise, not visits), and the
-  // analytics page itself (/_a), even though fetch-navigation treats it
-  // like a normal article.
+  // editor is open (body.editing — admin noise, not visits), and
+  // navigations TO the analytics page (/_a — admin machinery, and the
+  // server rejects it as a ping target anyway). Navigations AWAY from /_a
+  // must ping: load() already fetched the target page without the preload
+  // header, and without the ping that GET would flush to the crawler list.
   // Admins (when SSO is actually in use — with no auth proxy "admin" is
   // everyone's state) ping normally but with hide=1: the server then
   // records nothing and scrubs any session the same browser accumulated
@@ -395,7 +397,7 @@ import "overlayscrollbars/overlayscrollbars.css";
   // See docs/analytics.md.
   function ping(to, fr = currentPath, read = 0) {
     if (document.body.classList.contains("editing")) return;
-    if ((to && to === "/_a") || fr === "/_a") return;
+    if (to && to === "/_a") return;
     const hide = ssoAvailable && isAdmin ? 1 : 0;
     const body = JSON.stringify({
       fr, to, hide,

pagerite/analytics.py

@@ -316,6 +316,10 @@ class Store:
                 pass  # legacy schema / corrupt or unreadable file: start fresh
         #: client hash -> index of the current visit in data.visits
         self.sessions: dict[bytes, int] = {}
+        #: visit index -> count events recorded for that visit, so
+        #: ``_remove_visit`` can reverse all of them — not just the ones
+        #: from the visit's creation.  In-memory only, like ``sessions``.
+        self._count_log: dict[int, list[tuple]] = {}
         #: ip -> external https origin of the latest document GET carrying
         #: one, stashed for the visit the client's initial ping starts.
         #: Internal or absent referers never touch the table.
@@ -403,35 +407,44 @@ class Store:
                 del table[key]
 
     def _remove_visit(self, index: int) -> None:
-        """Delete a visit and reverse the counts its creation recorded.
+        """Delete a visit and reverse every count it recorded.
 
         Used when a known visitor turns out to be an admin (hide=1 ping):
-        the session is scrubbed from the stats.  Views/transitions logged
-        by later pings inside the visit lack per-event timestamps and are
-        left as-is.
+        the session is scrubbed from the stats.  The in-memory
+        ``_count_log`` tracks each site-visit/view/transition count the
+        visit produced, so the scrub reverses all of them — including the
+        ones logged by later pings inside the visit.
         """
-        visit = self.data.visits[index]
-        bucket = _bucket(visit.start)
-        self._uncount(self.data.site_visits, bucket)
-        views = self.data.views.get(visit.entry)
-        if views is not None:
-            self._uncount(views, bucket)
-            if not views:
-                del self.data.views[visit.entry]
-        fr_map = self.data.transitions.get(visit.referer or "(direct)")
-        if fr_map is not None:
-            buckets = fr_map.get(visit.entry)
-            if buckets is not None:
-                self._uncount(buckets, bucket)
-                if not buckets:
-                    del fr_map[visit.entry]
-            if not fr_map:
-                del self.data.transitions[visit.referer or "(direct)"]
+        for event in self._count_log.pop(index, ()):
+            kind = event[0]
+            if kind == "site":
+                self._uncount(self.data.site_visits, event[1])
+            elif kind == "view":
+                views = self.data.views.get(event[1])
+                if views is not None:
+                    self._uncount(views, event[2])
+                    if not views:
+                        del self.data.views[event[1]]
+            else:  # transition
+                _, fr, to, bucket = event
+                fr_map = self.data.transitions.get(fr)
+                if fr_map is not None:
+                    buckets = fr_map.get(to)
+                    if buckets is not None:
+                        self._uncount(buckets, bucket)
+                        if not buckets:
+                            del fr_map[to]
+                    if not fr_map:
+                        del self.data.transitions[fr]
         del self.data.visits[index]
-        # Sessions store list indices; shift the ones past the removed visit.
+        # Sessions and count logs store list indices; shift the ones past
+        # the removed visit.
         for key, i in list(self.sessions.items()):
             if i > index:
                 self.sessions[key] = i - 1
+        self._count_log = {
+            i - 1 if i > index else i: log for i, log in self._count_log.items()
+        }
 
     def _client_ip(self, client_hash: bytes) -> str:
         """Return the IP stored for ``client_hash``, or "" if missing."""
@@ -590,10 +603,18 @@ class Store:
         )
         visit.statuses[entry] = status
         self.data.visits.append(visit)
-        self.sessions[client_hash] = len(self.data.visits) - 1
-        self._count(self.data.site_visits, _bucket(now))
-        self._count(self.data.views.setdefault(entry, {}), _bucket(now))
-        self._count_transition(referer or "(direct)", entry, now)
+        index = len(self.data.visits) - 1
+        self.sessions[client_hash] = index
+        bucket = _bucket(now)
+        fr = referer or "(direct)"
+        self._count(self.data.site_visits, bucket)
+        self._count(self.data.views.setdefault(entry, {}), bucket)
+        self._count_transition(fr, entry, now)
+        self._count_log[index] = [
+            ("site", bucket),
+            ("view", entry, bucket),
+            ("transition", fr, entry, bucket),
+        ]
         return visit
 
     def track_entry(
@@ -763,9 +784,13 @@ class Store:
         else:
             visit = self.data.visits[index]
             now = datetime.now(UTC)
+            bucket = _bucket(now)
+            log = self._count_log.setdefault(index, [])
             if target.startswith("/"):
-                self._count(self.data.views.setdefault(target, {}), _bucket(now))
+                self._count(self.data.views.setdefault(target, {}), bucket)
+                log.append(("view", target, bucket))
             self._count_transition(fr, target, now)
+            log.append(("transition", fr, target, bucket))
             # First-seen only: repeat pages and repeated exits don't append.
             if visit.entry != target and target not in visit.trail:
                 visit.trail.append(target)