Raw access-log analytics storage with display-time classification

Replace the pre-classified store (visits/crawlers/abuse lists written at
collection time, plus abuse_ips and in-memory pending/session tables) with
a raw append-only log: one Get record per document GET (full path, true
HTTP status, referer origin, preload flag) and one Msg per /_ws activity
message.  Visitor/crawler/abuse classification, visit grouping (30-minute
inactivity gap), status/referer/UTM attribution and all aggregates are
derived in Store.display(), so future rule changes never invalidate stored
data.  The viewer payload keeps its exact shape.

Fixes structurally:

- Abuser 404s on slug-format paths showed up as "articles read": the
  not-found branch recorded the request twice through separate status
  plumbing.  Each request is now recorded once with its true status.
- 404 trail links never rendered red: cache-served navigations issue no
  GET and the only real GET (the idle preload) was discarded before status
  recording.  Preloads are now recorded with pre=True, never counted, and
  used for status attribution.
- formatAbuseRows merged a path's 404 probes and 200 reads into one entry;
  the collapse is now keyed by (path, status class).

Rule improvements enabled by the redesign:

- The plain-404 abuse threshold counts within a 1-hour sliding window, so
  long-time readers accumulating misses never classify (scanners spray).
- Hidden (admin) clients never trigger abuse classification: editing means
  visiting not-found pages.
- /.well-known/ probes (RFC 8615, e.g. Chrome devtools) are never abuse
  evidence; //foo-style empty path segments are an instant telltale.
- Visits can no longer open on an external exit URL; favicon fetches skip
  hidden clients' referers/exits.

Legacy analytics.json files are set aside as .bak-legacy on startup.
This commit is contained in:
2026-09-03 18:52:16 +00:00
parent 792b9e7aa9
commit eb2e8f8273
5 changed files with 789 additions and 797 deletions
+9 -4
View File
@@ -434,7 +434,9 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
/**
* Group abuse hits by IP and format each group as a row with the full paths
* probed. Identical paths are collapsed into one entry with their hit count.
* probed. Identical requests (same path and status class) are collapsed
* into one entry with their hit count; a path's 404 probes and its real
* (200) reads never merge.
* The paths split into two lists: ``paths`` holds the 404 probes (flagged
* paths — the ones that triggered abuse classification — first, then other
* 404s) shown verbatim, query string included, and ``articles`` holds the
@@ -465,7 +467,11 @@ export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) {
g.lastClient = a.client
}
const path = a.path || ''
const existing = g.pathCounts.get(path) || {
// Collapse identical requests, but never merge a path's 404 probes with
// its real (200) reads — a page probed while missing and later created
// must show up in both columns, not flip to "articles read".
const key = `${a.is_404 ? '4' : '2'}${path}`
const existing = g.pathCounts.get(key) || {
path,
count: 0,
firstStart: start,
@@ -475,8 +481,7 @@ export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) {
existing.count += 1
if (start < existing.firstStart) existing.firstStart = start
if (a.flag) existing.flag = true
if (!a.is_404) existing.is_404 = false
g.pathCounts.set(path, existing)
g.pathCounts.set(key, existing)
g.clientHashes.add(a.client)
groups.set(ip, g)
}