Reclassify sub-5s visits as crawlers, show crawler referers and abuser articles.

Visits with under 5 s of total reported reading time are JS-running bots:
display() converts them to crawler hits (one per trail page, with referer
and UTM query) and excludes them from every aggregate.  Crawler referers
join the favicon fetch origins and render with their icon in the crawler
table.  Abuse hits now record the real response status, so the abuse table
splits 404 probes from the articles the abuser actually read (200 GETs),
shown as trail links like the visitor/crawler tables.

docs/analytics.md

@@ -47,6 +47,9 @@ WebSocket reports actual navigations and active time spent on a page.
   (anything calling
   itself a "bot", plus known exceptions such as GoogleOther) are ignored
   server-side, and their document GETs land in the crawler list instead.
+  Real-browser bots whose UA does not match still register a visit, but
+  their reported reading time stays under 5 seconds, so they are
+  reclassified as crawler hits at display time (see **Crawler hits** below).
   No source-IP verification is done: a spoofed bot UA merely lands in the
   crawler stats, and scanners that probe telltale paths are caught by the
   abuse rules regardless. Reloads are not
@@ -90,7 +93,7 @@ WebSocket reports actual navigations and active time spent on a page.
   ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the
   https origin and accepted only when the client sent exactly that.
 - **External-site favicons**: for every external https origin seen as a visit
-  referer or an exit link, the server fetches `{origin}/favicon.ico` in a
+  referer, a crawler-hit referer or an exit link, the server fetches `{origin}/favicon.ico` in a
   background task (httpx, 8 s timeout, ≤ 64 KB, image content-types only —
   SVG is sniffed from the body when served without an image type) and stores
   the icon content-hashed on disk in the FileStore (served at `/_f/{name}`,
@@ -134,11 +137,19 @@ WebSocket reports actual navigations and active time spent on a page.
   from the same client arrives within 10 seconds the hit is discarded;
   otherwise it is written to `crawlers` — unless the client is hidden
   (admin), in which case the hit is discarded on expiry too.  Crawlers do not count as
-  visits or views.  The `Accept-Language` header is stored on the shared
+  visits or views.  Bots running real browsers can still slip past the UA
+  check: a visit whose total reported reading time stays under 5 seconds
+  (`_MIN_VISIT_READ`; durations are client-provided and trusted — such bots
+  report 0–2 s) is reclassified as crawler hits at display time, one hit
+  per internal trail page, and counts in no visit aggregate.  The
+  `Accept-Language` header is stored on the shared
   `Client` immediately; reverse-DNS host names and DB-IP geoip
   country/city are filled in asynchronously, just like for real visits.  In
   the analytics viewer, crawler hits are grouped by client hash and shown as
-  a trail of internal pages that crawler visited; the crawler table lists
+  a trail of internal pages that crawler visited, preceded by its referer
+  when there is one — spiders often advertise their own site as the
+  referer, and it is rendered with its favicon like visit referers (crawler
+  referers are included in the favicon fetch origins).  The crawler table lists
   the most recent crawler first, with the most active as a tie-breaker.
 - **Abuse (scanner) hits**: a 404 for a telltale path — any URL segment
   starting with a dot (`/.env`, `/.git/config`) or ending in `.php` —
@@ -152,9 +163,12 @@ WebSocket reports actual navigations and active time spent on a page.
   is persisted in the JSON file; the plain-404 counters are RAM-only.  In the
   viewer, abuse hits are grouped by IP (never by client/UA — scanners
   randomize theirs) in a separate "Abuse" table.  Identical paths are
-  collapsed into one entry with their hit count; flagged paths that
+  collapsed into one entry with their hit count.  The 404 probes ("paths
-  triggered classification are lifted to the top, followed by other 404s and
+  abused": flagged paths that triggered classification first, then other
-  then document GETs from the abuser.  Raw User-Agent strings are shown one
+  404s) are kept in a separate column from the real articles the abuser
+  actually read ("articles read": document GETs that returned 200, not the
+  404 fallback rendering — rendered as trail links like the visitor and
+  crawler tables, with the query string stripped).  Raw User-Agent strings are shown one
   per line with their occurrence counts, and the full lists are click-to-copy.
 
 ## Visits and sessions
@@ -219,14 +233,16 @@ Each `AbuseHit` record:
 - `client` — 6-byte blake3 hash referencing `Analytics.clients`,
 - `flag` — true for the path that triggered abuse classification (telltale
   path or the 404 that crossed the threshold),
-- `is_404` — true for 404 responses, false for document GETs from the
+- `is_404` — true for 404 responses (probed paths and 404-fallback document
-  abuser.
+  GETs), false for real (200) document GETs — articles the abuser read.
 
 Crawler hits are grouped by client hash in the analytics viewer; abuse hits
 are grouped by IP alone (resolved from the referenced `Client`).  In the
-Abuse table identical paths are collapsed with their counts; flagged paths
+Abuse table identical paths are collapsed with their counts, split into the
-that triggered classification are lifted to the top, followed by other 404s
+404 probes (flagged paths that triggered classification first, then other
-and then document GETs from the abuser.  Within each category paths are
+404s, shown verbatim) and the 200 document GETs shown as trail links in the
+separate articles column.
+Within each list paths are
 sorted by count descending, then by their earliest hit.
 
 In the visitor and crawler tables, internal paths that returned a 404 status
@@ -237,7 +253,9 @@ to tell misses from real pages at a glance.
 
 Aggregates are **not stored**; they are computed at display time by
 `Store.display()` from the visit records (entry + `navs` log), skipping
-hidden clients' visits. This is what allows a client to become hidden after
+hidden clients' visits and short visits reclassified as crawler hits
+(under `_MIN_VISIT_READ` seconds of total reported reading time). This is
+what allows a client to become hidden after
 navigations were already logged: no counts need reversing. The computed
 shapes, part of the WebSocket payload (`Display` struct alongside `visits`,
 `crawlers`, `abuse` and `clients`):

frontend/src/AnalyticsView.vue

@@ -163,7 +163,7 @@ const favicons = computed(() => data.value?.favicons || {})
 const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value))
 const crawlers = computed(() => rangeData.value?.crawlers || [])
 const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value))
-const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, now.value))
+const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, pageTree.value, now.value))
 
 </script>
 
@@ -244,6 +244,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
               <tbody>
                 <tr v-for="(c, i) in crawlerRows" :key="i">
                   <td class="trail">
+                    <TrailLink v-if="c.refererStep" :step="c.refererStep" :favicons="favicons" @close="$emit('close')" />
                     <TrailLink v-for="(s, si) in c.pages" :key="si" :step="s" :count="s.count" @close="$emit('close')" />
                   </td>
                   <VisitorCell
@@ -270,6 +271,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
               <thead>
                 <tr>
                   <th>paths abused</th>
+                  <th>articles read</th>
                   <th>visitor</th>
                   <th class="last-seen">last seen</th>
                 </tr>
@@ -286,6 +288,11 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
                       <small v-if="a.paths.length > ABUSE_MAX_LINES" class="muted">+{{ a.paths.length - ABUSE_MAX_LINES }} more</small>
                     </div>
                   </td>
+                  <td class="trail clickable-list"
+                      @click="copyList(a.allArticles, $event)">
+                    <TrailLink v-for="(s, si) in a.articles" :key="si" :step="s" :count="s.count" @close="$emit('close')" />
+                    <small v-if="!a.articles.length" class="muted">—</small>
+                  </td>
                   <VisitorCell
                     :ip="a.ip"
                     :ip-display="a.ipDisplay"

frontend/src/analytics/format.js

@@ -370,7 +370,9 @@ export function mainDomain(host, limit = 24) {
 /**
  * Group raw crawler hits by client hash and format each group as a row showing
  * every internal page that crawler visited.  Rows are sorted by most recent hit
- * first, with total hits as a tie-breaker.
+ * first, with total hits as a tie-breaker.  The group's ``refererStep`` is the
+ * latest external referer seen for the crawler — spiders often advertise
+ * their own site there — rendered with its favicon like visit referers.
  * ``clients`` maps client hashes to client records.
  */
 export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) {
@@ -382,10 +384,12 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
       clientHash: c.client,
       client,
       lastStart: 0,
+      referer: '',
       pages: new Map(),
     }
     const start = new Date(c.start).getTime()
     if (start > g.lastStart) g.lastStart = start
+    if (c.referer) g.referer = c.referer
     if (c.entry?.startsWith('/')) {
       const existing = g.pages.get(c.entry) || { count: 0, status: c.status || 200 }
       existing.count += 1
@@ -410,6 +414,7 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
         lastSeen: formatWhen(g.lastStart, now),
         lastSeenIso: formatWhenIso(g.lastStart),
         lastSeenLocal: formatWhenLocal(g.lastStart),
+        refererStep: stepOf(g.referer, titles),
         pages: [...g.pages.entries()]
           .sort((a, b) => b[1].count - a[1].count)
           .map(([path, info]) => ({ ...stepOf(path, titles), count: info.count, status: info.status })),
@@ -430,16 +435,19 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
 /**
  * Group abuse hits by IP and format each group as a row with the full paths
  * probed.  Identical paths are collapsed into one entry with their hit count.
- * Flagged paths (the ones that triggered abuse classification) are lifted to
+ * The paths split into two lists: ``paths`` holds the 404 probes (flagged
- * the top, followed by other 404s, then document GETs from the abuser.  Within
+ * paths — the ones that triggered abuse classification — first, then other
- * each category paths are sorted by count descending, then earliest first.
+ * 404s) shown verbatim, query string included, and ``articles`` holds the
+ * real (200) document GETs as trail steps resolved against the page tree
+ * (query string stripped), rendered like the visitor/crawler trails.  Within
+ * each list paths are sorted by count descending, then earliest first.
  * Rows are sorted by most recent hit first.  Visitor metadata comes from the
  * latest client hash seen for the IP; ``clientCount`` tells the visitor cell
- * how many distinct client variations the IP produced.  Paths are shown
+ * how many distinct client variations the IP produced.
- * verbatim (query string included), not resolved against the page tree.
  * ``clients`` maps client hashes to client records.
  */
-export function formatAbuseRows(abuse, clients, now = Date.now()) {
+export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) {
+  const titles = buildTitleMap(pageTree)
   const groups = new Map()
   for (const a of abuse || []) {
     const client = (clients || {})[a.client] || {}
@@ -481,13 +489,14 @@ export function formatAbuseRows(abuse, clients, now = Date.now()) {
     .sort((a, b) => b.lastStart - a.lastStart)
     .slice(0, 10)
     .map((g) => {
-      const pathCategory = (p) => (p.flag ? 0 : p.is_404 ? 1 : 2)
+      const all = [...g.pathCounts.values()]
-      const paths = [...g.pathCounts.values()].sort(
+      const byCount = (a, b) => b.count - a.count || a.firstStart - b.firstStart
-        (a, b) =>
+      const paths = all
-          pathCategory(a) - pathCategory(b) ||
+        .filter((p) => p.flag || p.is_404)
-          b.count - a.count ||
+        .sort((a, b) => (a.flag ? 0 : 1) - (b.flag ? 0 : 1) || byCount(a, b))
-          a.firstStart - b.firstStart,
+      const articles = all.filter((p) => !p.flag && !p.is_404).sort(byCount)
-      )
+      const pathList = (list) =>
+        list.map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path)).join('\n')
       const client = (clients || {})[g.lastClient] || {}
       const host = client.host || ''
       const isHost = !!host
@@ -501,9 +510,14 @@ export function formatAbuseRows(abuse, clients, now = Date.now()) {
           flag: p.flag,
           is_404: p.is_404,
         })),
-        allPaths: paths
+        allPaths: pathList(paths),
-          .map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path))
+        articles: articles
-          .join('\n'),
+          .map((p) => {
+            const step = stepOf(p.path.split('?')[0], titles)
+            return step ? { ...step, count: p.count } : null
+          })
+          .filter(Boolean),
+        allArticles: pathList(articles),
         clientCount: g.clientHashes.size,
         ip: client.ip || g.ip,
         ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip || g.ip) || client.ip || g.ip || '—',

pagerite/analytics.py

@@ -14,7 +14,11 @@ that only fetch documents end up in the crawler list).  JS-running crawlers
 (Googlebot, GoogleOther, Applebot, ...) do connect and send messages, but
 their UA gives them
 away (``_is_bot_ua``) and their messages are ignored, so they land in the
-crawler list too.  Idle-time link preloads from pagerite.js carry an
+crawler list too.  Bots whose UA does not match are caught by engagement:
+a visit whose total reported reading time is under ``_MIN_VISIT_READ``
+seconds is reclassified as crawler hits at display time (durations are
+client-provided and trusted — real-browser bots report 0–2 s), so it never
+counts in the visit aggregates either.  Idle-time link preloads from pagerite.js carry an
 ``x-pagerite-preload`` header and are not tracked at all — the navigation
 message sent when the user actually navigates does the counting.
 Admin clients send ``hide``: the client record is flagged ``hide``,
@@ -43,7 +47,7 @@ from collections.abc import Callable
 from contextlib import suppress
 from datetime import UTC, datetime, timedelta
 from pathlib import Path
-from urllib.parse import parse_qs, urlparse
+from urllib.parse import parse_qs, urlencode, urlparse
 
 import blake3
 import msgspec
@@ -200,8 +204,10 @@ class AbuseHit(msgspec.Struct, omit_defaults=True):
     Unlike crawler hits the full request path (query string included) is
     kept: the interesting part is exactly which paths were probed.
     ``flag`` marks the path that triggered classification; ``is_404``
-    distinguishes 404 responses from document GETs made by the abuser.
+    distinguishes 404 responses (probed paths and 404-fallback document
-    Client metadata is held in ``Analytics.clients`` keyed by ``client``.
+    GETs) from real 200 document GETs — the abuser actually reading
+    articles.  Client metadata is held in ``Analytics.clients`` keyed by
+    ``client``.
     """
 
     start: datetime
@@ -212,7 +218,7 @@ class AbuseHit(msgspec.Struct, omit_defaults=True):
     #: True when this path triggered abuse classification (telltale path
     #: or the 404 that crossed the threshold).
     flag: bool = False
-    #: True for 404 responses; false for document GETs from the abuser.
+    #: True for 404 responses; false for real (200) document GETs.
     is_404: bool = False
 
 
@@ -347,6 +353,11 @@ def _utm_tags(query: str) -> dict[str, str]:
 
 _CRAWLER_TIMEOUT = timedelta(seconds=10)
 
+#: Minimum total reported reading time (seconds, summed over the trail) for
+#: a session to count as a visit; shorter sessions are JS-running bots and
+#: are shown as crawler hits instead.
+_MIN_VISIT_READ = 5
+
 #: How long a failed favicon fetch suppresses retries for the same origin.
 _FAVICON_RETRY = timedelta(days=7)
 
@@ -505,16 +516,43 @@ class Store:
     def display(self) -> Display:
         """Build the viewer payload, excluding hidden clients.
 
+        Visits whose total reported reading time is under
+        ``_MIN_VISIT_READ`` seconds are JS-running bots, not readers: they
+        are converted to crawler hits (one per internal trail page) and left
+        out of the visit list and every aggregate.
         The aggregates (site visits, page views, transitions) are computed
         here from the visit records rather than stored, so a client that
         becomes hidden after navigations were already logged disappears
         from every statistic.  Internal-path navigations count as page
         views; external https targets are transitions only.
         """
-        visits = [v for v in self.data.visits if not self._hidden(v.client)]
+        visits: list[Visit] = []
+        crawlers = [h for h in self.data.crawlers if not self._hidden(h.client)]
+        for visit in self.data.visits:
+            if self._hidden(visit.client):
+                continue
+            if sum(item.read for item in visit.trail.values()) >= _MIN_VISIT_READ:
+                visits.append(visit)
+                continue
+            query = urlencode(visit.utm)
+            first = True
+            for t, item in visit.trail.items():
+                if not item.to.startswith("/"):
+                    continue
+                crawlers.append(
+                    CrawlerHit(
+                        start=t,
+                        entry=item.to,
+                        client=visit.client,
+                        referer=visit.referer if first else "",
+                        query=query if first else "",
+                        status=item.status,
+                    )
+                )
+                first = False
         display = Display(
             visits=visits,
-            crawlers=[h for h in self.data.crawlers if not self._hidden(h.client)],
+            crawlers=crawlers,
             abuse=[h for h in self.data.abuse if not self._hidden(h.client)],
             clients={h: c for h, c in self.data.clients.items() if not c.hide},
             favicons={
@@ -601,7 +639,9 @@ class Store:
     def favicon_origins_needed(self) -> list[str]:
         """External https origins seen in visits whose favicon needs fetching.
 
-        Covers visit referers and external exit targets (trail and navs).
+        Covers visit referers and external exit targets (trail and navs),
+        plus crawler-hit referers — spiders often advertise their own site
+        as the referer, so the icon identifies them in the crawler table.
         Origins with a stored icon, or a miss younger than
         ``_FAVICON_RETRY``, are skipped.
         """
@@ -613,6 +653,9 @@ class Store:
                 origin = _origin(target.to)
                 if origin is not None:
                     origins.add(origin)
+        for hit in self.data.crawlers:
+            if hit.referer:
+                origins.add(hit.referer)
         now = datetime.now(UTC)
         return [
             origin
@@ -673,6 +716,7 @@ class Store:
                         h.client,
                         h.entry + (f"?{h.query}" if h.query else ""),
                         start=h.start,
+                        is_404=h.status != 200,
                     )
             pending = [
                 h for h in self.pending_crawlers if self._client_ip(h.client) == ip
@@ -686,6 +730,7 @@ class Store:
                         h.client,
                         h.entry + (f"?{h.query}" if h.query else ""),
                         start=h.start,
+                        is_404=h.status != 200,
                     )
         self._abuse_hit(client_hash, path, flag=flag, is_404=is_404)
         self._save()
@@ -779,7 +824,7 @@ class Store:
         client_hash = self._ensure_client(ip, ua, lang, country=country)
         if ip in self.data.abuse_ips:
             flushed = self._flush_crawlers()
-            self._abuse_hit(client_hash, full_path, is_404=False, flag=False)
+            self._abuse_hit(client_hash, full_path, is_404=status != 200, flag=False)
             self._save()
             return flushed
         now = datetime.now(UTC)