Fix crawler misclassification from /_a and orphan counts on visit scrub
Two analytics corrections verified against the production capture: - pagerite.js suppressed pings with fr == '/_a', but fetch-navigation away from the analytics page had already GET-ed the target without the preload header; the orphaned pending hit then flushed to the crawler list, classifying a real user as a crawler. Navigations away from /_a now ping normally (the server rejects /_a as a target regardless, and admin noise is already handled by hide=1). - _remove_visit only reversed the visit's creation counts, leaving views/transitions from later pings behind as orphans on the graph with no matching row in the visitor table. An in-memory per-visit count log now tracks every count event, so an admin hide=1 scrub reverses the visit completely.
This commit is contained in:
+10
-7
@@ -45,16 +45,19 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
|
||||
back), so the exit URL is not necessarily the last trail entry. Outbound
|
||||
links are stored by full URL so several links to the same domain remain
|
||||
distinct.
|
||||
- **Excluded**: back/forward (popstate) navigations, navigation involving
|
||||
the analytics page itself (`/_a`), and everything while the user has the
|
||||
editor open (`body.editing`). Admin noise, not visits.
|
||||
- **Excluded**: back/forward (popstate) navigations, navigating *to* the
|
||||
analytics page (`/_a` — its GET is untracked, and the server rejects it
|
||||
as a ping target anyway), and everything while the user has the editor
|
||||
open (`body.editing`). Admin noise, not visits. Navigating *away* from
|
||||
`/_a` does ping: the fetch-navigation already GET-ed the target page
|
||||
without the preload header, and without the ping that GET would flush to
|
||||
the crawler list.
|
||||
- **Admins**: when SSO is in use and the session is known to be an admin,
|
||||
the client still pings but adds `hide=1`. The server then records
|
||||
nothing — and if the same client session already had a visit from before
|
||||
logging in, that visit is removed from the JSON along with the counts
|
||||
recorded when it was created (site visit, entry view, entry transition).
|
||||
Views/transitions logged by later pings inside such a visit lack
|
||||
per-event timestamps and are left as-is. With no auth proxy (dev/test)
|
||||
logging in, that visit is removed from the JSON along with every count
|
||||
it recorded — an in-memory per-visit log of count events makes full
|
||||
reversal possible. With no auth proxy (dev/test)
|
||||
"admin" is everyone's state, so `hide` stays 0 and everything is recorded.
|
||||
- The server validates `to`: internal paths must be valid slug paths
|
||||
("/" or `[a-z0-9_-]` segments), external ones are re-derived to the
|
||||
|
||||
@@ -385,9 +385,11 @@ import "overlayscrollbars/overlayscrollbars.css";
|
||||
// Reading time pauses after 1 minute of inactivity and resumes on the
|
||||
// next mouse/touch/scroll/keyboard event.
|
||||
// Excluded: back/forward (popstate never pings), everything while the
|
||||
// editor is open (body.editing — admin noise, not visits), and the
|
||||
// analytics page itself (/_a), even though fetch-navigation treats it
|
||||
// like a normal article.
|
||||
// editor is open (body.editing — admin noise, not visits), and
|
||||
// navigations TO the analytics page (/_a — admin machinery, and the
|
||||
// server rejects it as a ping target anyway). Navigations AWAY from /_a
|
||||
// must ping: load() already fetched the target page without the preload
|
||||
// header, and without the ping that GET would flush to the crawler list.
|
||||
// Admins (when SSO is actually in use — with no auth proxy "admin" is
|
||||
// everyone's state) ping normally but with hide=1: the server then
|
||||
// records nothing and scrubs any session the same browser accumulated
|
||||
@@ -395,7 +397,7 @@ import "overlayscrollbars/overlayscrollbars.css";
|
||||
// See docs/analytics.md.
|
||||
function ping(to, fr = currentPath, read = 0) {
|
||||
if (document.body.classList.contains("editing")) return;
|
||||
if ((to && to === "/_a") || fr === "/_a") return;
|
||||
if (to && to === "/_a") return;
|
||||
const hide = ssoAvailable && isAdmin ? 1 : 0;
|
||||
const body = JSON.stringify({
|
||||
fr, to, hide,
|
||||
|
||||
+52
-27
@@ -316,6 +316,10 @@ class Store:
|
||||
pass # legacy schema / corrupt or unreadable file: start fresh
|
||||
#: client hash -> index of the current visit in data.visits
|
||||
self.sessions: dict[bytes, int] = {}
|
||||
#: visit index -> count events recorded for that visit, so
|
||||
#: ``_remove_visit`` can reverse all of them — not just the ones
|
||||
#: from the visit's creation. In-memory only, like ``sessions``.
|
||||
self._count_log: dict[int, list[tuple]] = {}
|
||||
#: ip -> external https origin of the latest document GET carrying
|
||||
#: one, stashed for the visit the client's initial ping starts.
|
||||
#: Internal or absent referers never touch the table.
|
||||
@@ -403,35 +407,44 @@ class Store:
|
||||
del table[key]
|
||||
|
||||
def _remove_visit(self, index: int) -> None:
|
||||
"""Delete a visit and reverse the counts its creation recorded.
|
||||
"""Delete a visit and reverse every count it recorded.
|
||||
|
||||
Used when a known visitor turns out to be an admin (hide=1 ping):
|
||||
the session is scrubbed from the stats. Views/transitions logged
|
||||
by later pings inside the visit lack per-event timestamps and are
|
||||
left as-is.
|
||||
the session is scrubbed from the stats. The in-memory
|
||||
``_count_log`` tracks each site-visit/view/transition count the
|
||||
visit produced, so the scrub reverses all of them — including the
|
||||
ones logged by later pings inside the visit.
|
||||
"""
|
||||
visit = self.data.visits[index]
|
||||
bucket = _bucket(visit.start)
|
||||
self._uncount(self.data.site_visits, bucket)
|
||||
views = self.data.views.get(visit.entry)
|
||||
if views is not None:
|
||||
self._uncount(views, bucket)
|
||||
if not views:
|
||||
del self.data.views[visit.entry]
|
||||
fr_map = self.data.transitions.get(visit.referer or "(direct)")
|
||||
if fr_map is not None:
|
||||
buckets = fr_map.get(visit.entry)
|
||||
if buckets is not None:
|
||||
self._uncount(buckets, bucket)
|
||||
if not buckets:
|
||||
del fr_map[visit.entry]
|
||||
if not fr_map:
|
||||
del self.data.transitions[visit.referer or "(direct)"]
|
||||
for event in self._count_log.pop(index, ()):
|
||||
kind = event[0]
|
||||
if kind == "site":
|
||||
self._uncount(self.data.site_visits, event[1])
|
||||
elif kind == "view":
|
||||
views = self.data.views.get(event[1])
|
||||
if views is not None:
|
||||
self._uncount(views, event[2])
|
||||
if not views:
|
||||
del self.data.views[event[1]]
|
||||
else: # transition
|
||||
_, fr, to, bucket = event
|
||||
fr_map = self.data.transitions.get(fr)
|
||||
if fr_map is not None:
|
||||
buckets = fr_map.get(to)
|
||||
if buckets is not None:
|
||||
self._uncount(buckets, bucket)
|
||||
if not buckets:
|
||||
del fr_map[to]
|
||||
if not fr_map:
|
||||
del self.data.transitions[fr]
|
||||
del self.data.visits[index]
|
||||
# Sessions store list indices; shift the ones past the removed visit.
|
||||
# Sessions and count logs store list indices; shift the ones past
|
||||
# the removed visit.
|
||||
for key, i in list(self.sessions.items()):
|
||||
if i > index:
|
||||
self.sessions[key] = i - 1
|
||||
self._count_log = {
|
||||
i - 1 if i > index else i: log for i, log in self._count_log.items()
|
||||
}
|
||||
|
||||
def _client_ip(self, client_hash: bytes) -> str:
|
||||
"""Return the IP stored for ``client_hash``, or "" if missing."""
|
||||
@@ -590,10 +603,18 @@ class Store:
|
||||
)
|
||||
visit.statuses[entry] = status
|
||||
self.data.visits.append(visit)
|
||||
self.sessions[client_hash] = len(self.data.visits) - 1
|
||||
self._count(self.data.site_visits, _bucket(now))
|
||||
self._count(self.data.views.setdefault(entry, {}), _bucket(now))
|
||||
self._count_transition(referer or "(direct)", entry, now)
|
||||
index = len(self.data.visits) - 1
|
||||
self.sessions[client_hash] = index
|
||||
bucket = _bucket(now)
|
||||
fr = referer or "(direct)"
|
||||
self._count(self.data.site_visits, bucket)
|
||||
self._count(self.data.views.setdefault(entry, {}), bucket)
|
||||
self._count_transition(fr, entry, now)
|
||||
self._count_log[index] = [
|
||||
("site", bucket),
|
||||
("view", entry, bucket),
|
||||
("transition", fr, entry, bucket),
|
||||
]
|
||||
return visit
|
||||
|
||||
def track_entry(
|
||||
@@ -763,9 +784,13 @@ class Store:
|
||||
else:
|
||||
visit = self.data.visits[index]
|
||||
now = datetime.now(UTC)
|
||||
bucket = _bucket(now)
|
||||
log = self._count_log.setdefault(index, [])
|
||||
if target.startswith("/"):
|
||||
self._count(self.data.views.setdefault(target, {}), _bucket(now))
|
||||
self._count(self.data.views.setdefault(target, {}), bucket)
|
||||
log.append(("view", target, bucket))
|
||||
self._count_transition(fr, target, now)
|
||||
log.append(("transition", fr, target, bucket))
|
||||
# First-seen only: repeat pages and repeated exits don't append.
|
||||
if visit.entry != target and target not in visit.trail:
|
||||
visit.trail.append(target)
|
||||
|
||||
Reference in New Issue
Block a user