diff --git a/docs/analytics.md b/docs/analytics.md index 7b1ea1f..90f3d05 100644 --- a/docs/analytics.md +++ b/docs/analytics.md @@ -47,6 +47,9 @@ WebSocket reports actual navigations and active time spent on a page. (anything calling itself a "bot", plus known exceptions such as GoogleOther) are ignored server-side, and their document GETs land in the crawler list instead. + Real-browser bots whose UA does not match still register a visit, but + their reported reading time stays under 5 seconds, so they are + reclassified as crawler hits at display time (see **Crawler hits** below). No source-IP verification is done: a spoofed bot UA merely lands in the crawler stats, and scanners that probe telltale paths are caught by the abuse rules regardless. Reloads are not @@ -90,7 +93,7 @@ WebSocket reports actual navigations and active time spent on a page. ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the https origin and accepted only when the client sent exactly that. - **External-site favicons**: for every external https origin seen as a visit - referer or an exit link, the server fetches `{origin}/favicon.ico` in a + referer, a crawler-hit referer or an exit link, the server fetches `{origin}/favicon.ico` in a background task (httpx, 8 s timeout, ≤ 64 KB, image content-types only — SVG is sniffed from the body when served without an image type) and stores the icon content-hashed on disk in the FileStore (served at `/_f/{name}`, @@ -134,11 +137,19 @@ WebSocket reports actual navigations and active time spent on a page. from the same client arrives within 10 seconds the hit is discarded; otherwise it is written to `crawlers` — unless the client is hidden (admin), in which case the hit is discarded on expiry too. Crawlers do not count as - visits or views. The `Accept-Language` header is stored on the shared + visits or views. Bots running real browsers can still slip past the UA + check: a visit whose total reported reading time stays under 5 seconds + (`_MIN_VISIT_READ`; durations are client-provided and trusted — such bots + report 0–2 s) is reclassified as crawler hits at display time, one hit + per internal trail page, and counts in no visit aggregate. The + `Accept-Language` header is stored on the shared `Client` immediately; reverse-DNS host names and DB-IP geoip country/city are filled in asynchronously, just like for real visits. In the analytics viewer, crawler hits are grouped by client hash and shown as - a trail of internal pages that crawler visited; the crawler table lists + a trail of internal pages that crawler visited, preceded by its referer + when there is one — spiders often advertise their own site as the + referer, and it is rendered with its favicon like visit referers (crawler + referers are included in the favicon fetch origins). The crawler table lists the most recent crawler first, with the most active as a tie-breaker. - **Abuse (scanner) hits**: a 404 for a telltale path — any URL segment starting with a dot (`/.env`, `/.git/config`) or ending in `.php` — @@ -152,9 +163,12 @@ WebSocket reports actual navigations and active time spent on a page. is persisted in the JSON file; the plain-404 counters are RAM-only. In the viewer, abuse hits are grouped by IP (never by client/UA — scanners randomize theirs) in a separate "Abuse" table. Identical paths are - collapsed into one entry with their hit count; flagged paths that - triggered classification are lifted to the top, followed by other 404s and - then document GETs from the abuser. Raw User-Agent strings are shown one + collapsed into one entry with their hit count. The 404 probes ("paths + abused": flagged paths that triggered classification first, then other + 404s) are kept in a separate column from the real articles the abuser + actually read ("articles read": document GETs that returned 200, not the + 404 fallback rendering — rendered as trail links like the visitor and + crawler tables, with the query string stripped). Raw User-Agent strings are shown one per line with their occurrence counts, and the full lists are click-to-copy. ## Visits and sessions @@ -219,14 +233,16 @@ Each `AbuseHit` record: - `client` — 6-byte blake3 hash referencing `Analytics.clients`, - `flag` — true for the path that triggered abuse classification (telltale path or the 404 that crossed the threshold), -- `is_404` — true for 404 responses, false for document GETs from the - abuser. +- `is_404` — true for 404 responses (probed paths and 404-fallback document + GETs), false for real (200) document GETs — articles the abuser read. Crawler hits are grouped by client hash in the analytics viewer; abuse hits are grouped by IP alone (resolved from the referenced `Client`). In the -Abuse table identical paths are collapsed with their counts; flagged paths -that triggered classification are lifted to the top, followed by other 404s -and then document GETs from the abuser. Within each category paths are +Abuse table identical paths are collapsed with their counts, split into the +404 probes (flagged paths that triggered classification first, then other +404s, shown verbatim) and the 200 document GETs shown as trail links in the +separate articles column. +Within each list paths are sorted by count descending, then by their earliest hit. In the visitor and crawler tables, internal paths that returned a 404 status @@ -237,7 +253,9 @@ to tell misses from real pages at a glance. Aggregates are **not stored**; they are computed at display time by `Store.display()` from the visit records (entry + `navs` log), skipping -hidden clients' visits. This is what allows a client to become hidden after +hidden clients' visits and short visits reclassified as crawler hits +(under `_MIN_VISIT_READ` seconds of total reported reading time). This is +what allows a client to become hidden after navigations were already logged: no counts need reversing. The computed shapes, part of the WebSocket payload (`Display` struct alongside `visits`, `crawlers`, `abuse` and `clients`): diff --git a/frontend/src/AnalyticsView.vue b/frontend/src/AnalyticsView.vue index 6cb69e3..144225c 100644 --- a/frontend/src/AnalyticsView.vue +++ b/frontend/src/AnalyticsView.vue @@ -163,7 +163,7 @@ const favicons = computed(() => data.value?.favicons || {}) const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value)) const crawlers = computed(() => rangeData.value?.crawlers || []) const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value)) -const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, now.value)) +const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, pageTree.value, now.value)) @@ -244,6 +244,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c + formatAbuseRows(rangeData.value?.abuse || [], c paths abused + articles read visitor last seen @@ -286,6 +288,11 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c +{{ a.paths.length - ABUSE_MAX_LINES }} more + + + — + g.lastStart) g.lastStart = start + if (c.referer) g.referer = c.referer if (c.entry?.startsWith('/')) { const existing = g.pages.get(c.entry) || { count: 0, status: c.status || 200 } existing.count += 1 @@ -410,6 +414,7 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) lastSeen: formatWhen(g.lastStart, now), lastSeenIso: formatWhenIso(g.lastStart), lastSeenLocal: formatWhenLocal(g.lastStart), + refererStep: stepOf(g.referer, titles), pages: [...g.pages.entries()] .sort((a, b) => b[1].count - a[1].count) .map(([path, info]) => ({ ...stepOf(path, titles), count: info.count, status: info.status })), @@ -430,16 +435,19 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) /** * Group abuse hits by IP and format each group as a row with the full paths * probed. Identical paths are collapsed into one entry with their hit count. - * Flagged paths (the ones that triggered abuse classification) are lifted to - * the top, followed by other 404s, then document GETs from the abuser. Within - * each category paths are sorted by count descending, then earliest first. + * The paths split into two lists: ``paths`` holds the 404 probes (flagged + * paths — the ones that triggered abuse classification — first, then other + * 404s) shown verbatim, query string included, and ``articles`` holds the + * real (200) document GETs as trail steps resolved against the page tree + * (query string stripped), rendered like the visitor/crawler trails. Within + * each list paths are sorted by count descending, then earliest first. * Rows are sorted by most recent hit first. Visitor metadata comes from the * latest client hash seen for the IP; ``clientCount`` tells the visitor cell - * how many distinct client variations the IP produced. Paths are shown - * verbatim (query string included), not resolved against the page tree. + * how many distinct client variations the IP produced. * ``clients`` maps client hashes to client records. */ -export function formatAbuseRows(abuse, clients, now = Date.now()) { +export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) { + const titles = buildTitleMap(pageTree) const groups = new Map() for (const a of abuse || []) { const client = (clients || {})[a.client] || {} @@ -481,13 +489,14 @@ export function formatAbuseRows(abuse, clients, now = Date.now()) { .sort((a, b) => b.lastStart - a.lastStart) .slice(0, 10) .map((g) => { - const pathCategory = (p) => (p.flag ? 0 : p.is_404 ? 1 : 2) - const paths = [...g.pathCounts.values()].sort( - (a, b) => - pathCategory(a) - pathCategory(b) || - b.count - a.count || - a.firstStart - b.firstStart, - ) + const all = [...g.pathCounts.values()] + const byCount = (a, b) => b.count - a.count || a.firstStart - b.firstStart + const paths = all + .filter((p) => p.flag || p.is_404) + .sort((a, b) => (a.flag ? 0 : 1) - (b.flag ? 0 : 1) || byCount(a, b)) + const articles = all.filter((p) => !p.flag && !p.is_404).sort(byCount) + const pathList = (list) => + list.map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path)).join('\n') const client = (clients || {})[g.lastClient] || {} const host = client.host || '' const isHost = !!host @@ -501,9 +510,14 @@ export function formatAbuseRows(abuse, clients, now = Date.now()) { flag: p.flag, is_404: p.is_404, })), - allPaths: paths - .map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path)) - .join('\n'), + allPaths: pathList(paths), + articles: articles + .map((p) => { + const step = stepOf(p.path.split('?')[0], titles) + return step ? { ...step, count: p.count } : null + }) + .filter(Boolean), + allArticles: pathList(articles), clientCount: g.clientHashes.size, ip: client.ip || g.ip, ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip || g.ip) || client.ip || g.ip || '—', diff --git a/pagerite/analytics.py b/pagerite/analytics.py index cf4887e..5ccf7d3 100644 --- a/pagerite/analytics.py +++ b/pagerite/analytics.py @@ -14,7 +14,11 @@ that only fetch documents end up in the crawler list). JS-running crawlers (Googlebot, GoogleOther, Applebot, ...) do connect and send messages, but their UA gives them away (``_is_bot_ua``) and their messages are ignored, so they land in the -crawler list too. Idle-time link preloads from pagerite.js carry an +crawler list too. Bots whose UA does not match are caught by engagement: +a visit whose total reported reading time is under ``_MIN_VISIT_READ`` +seconds is reclassified as crawler hits at display time (durations are +client-provided and trusted — real-browser bots report 0–2 s), so it never +counts in the visit aggregates either. Idle-time link preloads from pagerite.js carry an ``x-pagerite-preload`` header and are not tracked at all — the navigation message sent when the user actually navigates does the counting. Admin clients send ``hide``: the client record is flagged ``hide``, @@ -43,7 +47,7 @@ from collections.abc import Callable from contextlib import suppress from datetime import UTC, datetime, timedelta from pathlib import Path -from urllib.parse import parse_qs, urlparse +from urllib.parse import parse_qs, urlencode, urlparse import blake3 import msgspec @@ -200,8 +204,10 @@ class AbuseHit(msgspec.Struct, omit_defaults=True): Unlike crawler hits the full request path (query string included) is kept: the interesting part is exactly which paths were probed. ``flag`` marks the path that triggered classification; ``is_404`` - distinguishes 404 responses from document GETs made by the abuser. - Client metadata is held in ``Analytics.clients`` keyed by ``client``. + distinguishes 404 responses (probed paths and 404-fallback document + GETs) from real 200 document GETs — the abuser actually reading + articles. Client metadata is held in ``Analytics.clients`` keyed by + ``client``. """ start: datetime @@ -212,7 +218,7 @@ class AbuseHit(msgspec.Struct, omit_defaults=True): #: True when this path triggered abuse classification (telltale path #: or the 404 that crossed the threshold). flag: bool = False - #: True for 404 responses; false for document GETs from the abuser. + #: True for 404 responses; false for real (200) document GETs. is_404: bool = False @@ -347,6 +353,11 @@ def _utm_tags(query: str) -> dict[str, str]: _CRAWLER_TIMEOUT = timedelta(seconds=10) +#: Minimum total reported reading time (seconds, summed over the trail) for +#: a session to count as a visit; shorter sessions are JS-running bots and +#: are shown as crawler hits instead. +_MIN_VISIT_READ = 5 + #: How long a failed favicon fetch suppresses retries for the same origin. _FAVICON_RETRY = timedelta(days=7) @@ -505,16 +516,43 @@ class Store: def display(self) -> Display: """Build the viewer payload, excluding hidden clients. + Visits whose total reported reading time is under + ``_MIN_VISIT_READ`` seconds are JS-running bots, not readers: they + are converted to crawler hits (one per internal trail page) and left + out of the visit list and every aggregate. The aggregates (site visits, page views, transitions) are computed here from the visit records rather than stored, so a client that becomes hidden after navigations were already logged disappears from every statistic. Internal-path navigations count as page views; external https targets are transitions only. """ - visits = [v for v in self.data.visits if not self._hidden(v.client)] + visits: list[Visit] = [] + crawlers = [h for h in self.data.crawlers if not self._hidden(h.client)] + for visit in self.data.visits: + if self._hidden(visit.client): + continue + if sum(item.read for item in visit.trail.values()) >= _MIN_VISIT_READ: + visits.append(visit) + continue + query = urlencode(visit.utm) + first = True + for t, item in visit.trail.items(): + if not item.to.startswith("/"): + continue + crawlers.append( + CrawlerHit( + start=t, + entry=item.to, + client=visit.client, + referer=visit.referer if first else "", + query=query if first else "", + status=item.status, + ) + ) + first = False display = Display( visits=visits, - crawlers=[h for h in self.data.crawlers if not self._hidden(h.client)], + crawlers=crawlers, abuse=[h for h in self.data.abuse if not self._hidden(h.client)], clients={h: c for h, c in self.data.clients.items() if not c.hide}, favicons={ @@ -601,7 +639,9 @@ class Store: def favicon_origins_needed(self) -> list[str]: """External https origins seen in visits whose favicon needs fetching. - Covers visit referers and external exit targets (trail and navs). + Covers visit referers and external exit targets (trail and navs), + plus crawler-hit referers — spiders often advertise their own site + as the referer, so the icon identifies them in the crawler table. Origins with a stored icon, or a miss younger than ``_FAVICON_RETRY``, are skipped. """ @@ -613,6 +653,9 @@ class Store: origin = _origin(target.to) if origin is not None: origins.add(origin) + for hit in self.data.crawlers: + if hit.referer: + origins.add(hit.referer) now = datetime.now(UTC) return [ origin @@ -673,6 +716,7 @@ class Store: h.client, h.entry + (f"?{h.query}" if h.query else ""), start=h.start, + is_404=h.status != 200, ) pending = [ h for h in self.pending_crawlers if self._client_ip(h.client) == ip @@ -686,6 +730,7 @@ class Store: h.client, h.entry + (f"?{h.query}" if h.query else ""), start=h.start, + is_404=h.status != 200, ) self._abuse_hit(client_hash, path, flag=flag, is_404=is_404) self._save() @@ -779,7 +824,7 @@ class Store: client_hash = self._ensure_client(ip, ua, lang, country=country) if ip in self.data.abuse_ips: flushed = self._flush_crawlers() - self._abuse_hit(client_hash, full_path, is_404=False, flag=False) + self._abuse_hit(client_hash, full_path, is_404=status != 200, flag=False) self._save() return flushed now = datetime.now(UTC)