From 85ef968cd63cd2c0569badd16a0a7d4e2043e321 Mon Sep 17 00:00:00 2001 From: Leo Vasanko Date: Thu, 3 Sep 2026 15:43:43 +0000 Subject: [PATCH] Reclassify sub-5s visits as crawlers, show crawler referers and abuser articles. Visits with under 5 s of total reported reading time are JS-running bots: display() converts them to crawler hits (one per trail page, with referer and UTM query) and excludes them from every aggregate. Crawler referers join the favicon fetch origins and render with their icon in the crawler table. Abuse hits now record the real response status, so the abuse table splits 404 probes from the articles the abuser actually read (200 GETs), shown as trail links like the visitor/crawler tables. --- docs/analytics.md | 42 +++++++++++++++------ frontend/src/AnalyticsView.vue | 9 ++++- frontend/src/analytics/format.js | 48 +++++++++++++++--------- pagerite/analytics.py | 63 +++++++++++++++++++++++++++----- 4 files changed, 123 insertions(+), 39 deletions(-) diff --git a/docs/analytics.md b/docs/analytics.md index 7b1ea1f..90f3d05 100644 --- a/docs/analytics.md +++ b/docs/analytics.md @@ -47,6 +47,9 @@ WebSocket reports actual navigations and active time spent on a page. (anything calling itself a "bot", plus known exceptions such as GoogleOther) are ignored server-side, and their document GETs land in the crawler list instead. + Real-browser bots whose UA does not match still register a visit, but + their reported reading time stays under 5 seconds, so they are + reclassified as crawler hits at display time (see **Crawler hits** below). No source-IP verification is done: a spoofed bot UA merely lands in the crawler stats, and scanners that probe telltale paths are caught by the abuse rules regardless. Reloads are not @@ -90,7 +93,7 @@ WebSocket reports actual navigations and active time spent on a page. ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the https origin and accepted only when the client sent exactly that. - **External-site favicons**: for every external https origin seen as a visit - referer or an exit link, the server fetches `{origin}/favicon.ico` in a + referer, a crawler-hit referer or an exit link, the server fetches `{origin}/favicon.ico` in a background task (httpx, 8 s timeout, ≤ 64 KB, image content-types only — SVG is sniffed from the body when served without an image type) and stores the icon content-hashed on disk in the FileStore (served at `/_f/{name}`, @@ -134,11 +137,19 @@ WebSocket reports actual navigations and active time spent on a page. from the same client arrives within 10 seconds the hit is discarded; otherwise it is written to `crawlers` — unless the client is hidden (admin), in which case the hit is discarded on expiry too. Crawlers do not count as - visits or views. The `Accept-Language` header is stored on the shared + visits or views. Bots running real browsers can still slip past the UA + check: a visit whose total reported reading time stays under 5 seconds + (`_MIN_VISIT_READ`; durations are client-provided and trusted — such bots + report 0–2 s) is reclassified as crawler hits at display time, one hit + per internal trail page, and counts in no visit aggregate. The + `Accept-Language` header is stored on the shared `Client` immediately; reverse-DNS host names and DB-IP geoip country/city are filled in asynchronously, just like for real visits. In the analytics viewer, crawler hits are grouped by client hash and shown as - a trail of internal pages that crawler visited; the crawler table lists + a trail of internal pages that crawler visited, preceded by its referer + when there is one — spiders often advertise their own site as the + referer, and it is rendered with its favicon like visit referers (crawler + referers are included in the favicon fetch origins). The crawler table lists the most recent crawler first, with the most active as a tie-breaker. - **Abuse (scanner) hits**: a 404 for a telltale path — any URL segment starting with a dot (`/.env`, `/.git/config`) or ending in `.php` — @@ -152,9 +163,12 @@ WebSocket reports actual navigations and active time spent on a page. is persisted in the JSON file; the plain-404 counters are RAM-only. In the viewer, abuse hits are grouped by IP (never by client/UA — scanners randomize theirs) in a separate "Abuse" table. Identical paths are - collapsed into one entry with their hit count; flagged paths that - triggered classification are lifted to the top, followed by other 404s and - then document GETs from the abuser. Raw User-Agent strings are shown one + collapsed into one entry with their hit count. The 404 probes ("paths + abused": flagged paths that triggered classification first, then other + 404s) are kept in a separate column from the real articles the abuser + actually read ("articles read": document GETs that returned 200, not the + 404 fallback rendering — rendered as trail links like the visitor and + crawler tables, with the query string stripped). Raw User-Agent strings are shown one per line with their occurrence counts, and the full lists are click-to-copy. ## Visits and sessions @@ -219,14 +233,16 @@ Each `AbuseHit` record: - `client` — 6-byte blake3 hash referencing `Analytics.clients`, - `flag` — true for the path that triggered abuse classification (telltale path or the 404 that crossed the threshold), -- `is_404` — true for 404 responses, false for document GETs from the - abuser. +- `is_404` — true for 404 responses (probed paths and 404-fallback document + GETs), false for real (200) document GETs — articles the abuser read. Crawler hits are grouped by client hash in the analytics viewer; abuse hits are grouped by IP alone (resolved from the referenced `Client`). In the -Abuse table identical paths are collapsed with their counts; flagged paths -that triggered classification are lifted to the top, followed by other 404s -and then document GETs from the abuser. Within each category paths are +Abuse table identical paths are collapsed with their counts, split into the +404 probes (flagged paths that triggered classification first, then other +404s, shown verbatim) and the 200 document GETs shown as trail links in the +separate articles column. +Within each list paths are sorted by count descending, then by their earliest hit. In the visitor and crawler tables, internal paths that returned a 404 status @@ -237,7 +253,9 @@ to tell misses from real pages at a glance. Aggregates are **not stored**; they are computed at display time by `Store.display()` from the visit records (entry + `navs` log), skipping -hidden clients' visits. This is what allows a client to become hidden after +hidden clients' visits and short visits reclassified as crawler hits +(under `_MIN_VISIT_READ` seconds of total reported reading time). This is +what allows a client to become hidden after navigations were already logged: no counts need reversing. The computed shapes, part of the WebSocket payload (`Display` struct alongside `visits`, `crawlers`, `abuse` and `clients`): diff --git a/frontend/src/AnalyticsView.vue b/frontend/src/AnalyticsView.vue index 6cb69e3..144225c 100644 --- a/frontend/src/AnalyticsView.vue +++ b/frontend/src/AnalyticsView.vue @@ -163,7 +163,7 @@ const favicons = computed(() => data.value?.favicons || {}) const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value)) const crawlers = computed(() => rangeData.value?.crawlers || []) const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value)) -const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, now.value)) +const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, pageTree.value, now.value)) @@ -244,6 +244,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c + formatAbuseRows(rangeData.value?.abuse || [], c paths abused + articles read visitor last seen @@ -286,6 +288,11 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c +{{ a.paths.length - ABUSE_MAX_LINES }} more + + + — + g.lastStart) g.lastStart = start + if (c.referer) g.referer = c.referer if (c.entry?.startsWith('/')) { const existing = g.pages.get(c.entry) || { count: 0, status: c.status || 200 } existing.count += 1 @@ -410,6 +414,7 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) lastSeen: formatWhen(g.lastStart, now), lastSeenIso: formatWhenIso(g.lastStart), lastSeenLocal: formatWhenLocal(g.lastStart), + refererStep: stepOf(g.referer, titles), pages: [...g.pages.entries()] .sort((a, b) => b[1].count - a[1].count) .map(([path, info]) => ({ ...stepOf(path, titles), count: info.count, status: info.status })), @@ -430,16 +435,19 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) /** * Group abuse hits by IP and format each group as a row with the full paths * probed. Identical paths are collapsed into one entry with their hit count. - * Flagged paths (the ones that triggered abuse classification) are lifted to - * the top, followed by other 404s, then document GETs from the abuser. Within - * each category paths are sorted by count descending, then earliest first. + * The paths split into two lists: ``paths`` holds the 404 probes (flagged + * paths — the ones that triggered abuse classification — first, then other + * 404s) shown verbatim, query string included, and ``articles`` holds the + * real (200) document GETs as trail steps resolved against the page tree + * (query string stripped), rendered like the visitor/crawler trails. Within + * each list paths are sorted by count descending, then earliest first. * Rows are sorted by most recent hit first. Visitor metadata comes from the * latest client hash seen for the IP; ``clientCount`` tells the visitor cell - * how many distinct client variations the IP produced. Paths are shown - * verbatim (query string included), not resolved against the page tree. + * how many distinct client variations the IP produced. * ``clients`` maps client hashes to client records. */ -export function formatAbuseRows(abuse, clients, now = Date.now()) { +export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) { + const titles = buildTitleMap(pageTree) const groups = new Map() for (const a of abuse || []) { const client = (clients || {})[a.client] || {} @@ -481,13 +489,14 @@ export function formatAbuseRows(abuse, clients, now = Date.now()) { .sort((a, b) => b.lastStart - a.lastStart) .slice(0, 10) .map((g) => { - const pathCategory = (p) => (p.flag ? 0 : p.is_404 ? 1 : 2) - const paths = [...g.pathCounts.values()].sort( - (a, b) => - pathCategory(a) - pathCategory(b) || - b.count - a.count || - a.firstStart - b.firstStart, - ) + const all = [...g.pathCounts.values()] + const byCount = (a, b) => b.count - a.count || a.firstStart - b.firstStart + const paths = all + .filter((p) => p.flag || p.is_404) + .sort((a, b) => (a.flag ? 0 : 1) - (b.flag ? 0 : 1) || byCount(a, b)) + const articles = all.filter((p) => !p.flag && !p.is_404).sort(byCount) + const pathList = (list) => + list.map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path)).join('\n') const client = (clients || {})[g.lastClient] || {} const host = client.host || '' const isHost = !!host @@ -501,9 +510,14 @@ export function formatAbuseRows(abuse, clients, now = Date.now()) { flag: p.flag, is_404: p.is_404, })), - allPaths: paths - .map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path)) - .join('\n'), + allPaths: pathList(paths), + articles: articles + .map((p) => { + const step = stepOf(p.path.split('?')[0], titles) + return step ? { ...step, count: p.count } : null + }) + .filter(Boolean), + allArticles: pathList(articles), clientCount: g.clientHashes.size, ip: client.ip || g.ip, ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip || g.ip) || client.ip || g.ip || '—', diff --git a/pagerite/analytics.py b/pagerite/analytics.py index cf4887e..5ccf7d3 100644 --- a/pagerite/analytics.py +++ b/pagerite/analytics.py @@ -14,7 +14,11 @@ that only fetch documents end up in the crawler list). JS-running crawlers (Googlebot, GoogleOther, Applebot, ...) do connect and send messages, but their UA gives them away (``_is_bot_ua``) and their messages are ignored, so they land in the -crawler list too. Idle-time link preloads from pagerite.js carry an +crawler list too. Bots whose UA does not match are caught by engagement: +a visit whose total reported reading time is under ``_MIN_VISIT_READ`` +seconds is reclassified as crawler hits at display time (durations are +client-provided and trusted — real-browser bots report 0–2 s), so it never +counts in the visit aggregates either. Idle-time link preloads from pagerite.js carry an ``x-pagerite-preload`` header and are not tracked at all — the navigation message sent when the user actually navigates does the counting. Admin clients send ``hide``: the client record is flagged ``hide``, @@ -43,7 +47,7 @@ from collections.abc import Callable from contextlib import suppress from datetime import UTC, datetime, timedelta from pathlib import Path -from urllib.parse import parse_qs, urlparse +from urllib.parse import parse_qs, urlencode, urlparse import blake3 import msgspec @@ -200,8 +204,10 @@ class AbuseHit(msgspec.Struct, omit_defaults=True): Unlike crawler hits the full request path (query string included) is kept: the interesting part is exactly which paths were probed. ``flag`` marks the path that triggered classification; ``is_404`` - distinguishes 404 responses from document GETs made by the abuser. - Client metadata is held in ``Analytics.clients`` keyed by ``client``. + distinguishes 404 responses (probed paths and 404-fallback document + GETs) from real 200 document GETs — the abuser actually reading + articles. Client metadata is held in ``Analytics.clients`` keyed by + ``client``. """ start: datetime @@ -212,7 +218,7 @@ class AbuseHit(msgspec.Struct, omit_defaults=True): #: True when this path triggered abuse classification (telltale path #: or the 404 that crossed the threshold). flag: bool = False - #: True for 404 responses; false for document GETs from the abuser. + #: True for 404 responses; false for real (200) document GETs. is_404: bool = False @@ -347,6 +353,11 @@ def _utm_tags(query: str) -> dict[str, str]: _CRAWLER_TIMEOUT = timedelta(seconds=10) +#: Minimum total reported reading time (seconds, summed over the trail) for +#: a session to count as a visit; shorter sessions are JS-running bots and +#: are shown as crawler hits instead. +_MIN_VISIT_READ = 5 + #: How long a failed favicon fetch suppresses retries for the same origin. _FAVICON_RETRY = timedelta(days=7) @@ -505,16 +516,43 @@ class Store: def display(self) -> Display: """Build the viewer payload, excluding hidden clients. + Visits whose total reported reading time is under + ``_MIN_VISIT_READ`` seconds are JS-running bots, not readers: they + are converted to crawler hits (one per internal trail page) and left + out of the visit list and every aggregate. The aggregates (site visits, page views, transitions) are computed here from the visit records rather than stored, so a client that becomes hidden after navigations were already logged disappears from every statistic. Internal-path navigations count as page views; external https targets are transitions only. """ - visits = [v for v in self.data.visits if not self._hidden(v.client)] + visits: list[Visit] = [] + crawlers = [h for h in self.data.crawlers if not self._hidden(h.client)] + for visit in self.data.visits: + if self._hidden(visit.client): + continue + if sum(item.read for item in visit.trail.values()) >= _MIN_VISIT_READ: + visits.append(visit) + continue + query = urlencode(visit.utm) + first = True + for t, item in visit.trail.items(): + if not item.to.startswith("/"): + continue + crawlers.append( + CrawlerHit( + start=t, + entry=item.to, + client=visit.client, + referer=visit.referer if first else "", + query=query if first else "", + status=item.status, + ) + ) + first = False display = Display( visits=visits, - crawlers=[h for h in self.data.crawlers if not self._hidden(h.client)], + crawlers=crawlers, abuse=[h for h in self.data.abuse if not self._hidden(h.client)], clients={h: c for h, c in self.data.clients.items() if not c.hide}, favicons={ @@ -601,7 +639,9 @@ class Store: def favicon_origins_needed(self) -> list[str]: """External https origins seen in visits whose favicon needs fetching. - Covers visit referers and external exit targets (trail and navs). + Covers visit referers and external exit targets (trail and navs), + plus crawler-hit referers — spiders often advertise their own site + as the referer, so the icon identifies them in the crawler table. Origins with a stored icon, or a miss younger than ``_FAVICON_RETRY``, are skipped. """ @@ -613,6 +653,9 @@ class Store: origin = _origin(target.to) if origin is not None: origins.add(origin) + for hit in self.data.crawlers: + if hit.referer: + origins.add(hit.referer) now = datetime.now(UTC) return [ origin @@ -673,6 +716,7 @@ class Store: h.client, h.entry + (f"?{h.query}" if h.query else ""), start=h.start, + is_404=h.status != 200, ) pending = [ h for h in self.pending_crawlers if self._client_ip(h.client) == ip @@ -686,6 +730,7 @@ class Store: h.client, h.entry + (f"?{h.query}" if h.query else ""), start=h.start, + is_404=h.status != 200, ) self._abuse_hit(client_hash, path, flag=flag, is_404=is_404) self._save() @@ -779,7 +824,7 @@ class Store: client_hash = self._ensure_client(ip, ua, lang, country=country) if ip in self.data.abuse_ips: flushed = self._flush_crawlers() - self._abuse_hit(client_hash, full_path, is_404=False, flag=False) + self._abuse_hit(client_hash, full_path, is_404=status != 200, flag=False) self._save() return flushed now = datetime.now(UTC)