Raw access-log analytics storage with display-time classification
Replace the pre-classified store (visits/crawlers/abuse lists written at collection time, plus abuse_ips and in-memory pending/session tables) with a raw append-only log: one Get record per document GET (full path, true HTTP status, referer origin, preload flag) and one Msg per /_ws activity message. Visitor/crawler/abuse classification, visit grouping (30-minute inactivity gap), status/referer/UTM attribution and all aggregates are derived in Store.display(), so future rule changes never invalidate stored data. The viewer payload keeps its exact shape. Fixes structurally: - Abuser 404s on slug-format paths showed up as "articles read": the not-found branch recorded the request twice through separate status plumbing. Each request is now recorded once with its true status. - 404 trail links never rendered red: cache-served navigations issue no GET and the only real GET (the idle preload) was discarded before status recording. Preloads are now recorded with pre=True, never counted, and used for status attribution. - formatAbuseRows merged a path's 404 probes and 200 reads into one entry; the collapse is now keyed by (path, status class). Rule improvements enabled by the redesign: - The plain-404 abuse threshold counts within a 1-hour sliding window, so long-time readers accumulating misses never classify (scanners spray). - Hidden (admin) clients never trigger abuse classification: editing means visiting not-found pages. - /.well-known/ probes (RFC 8615, e.g. Chrome devtools) are never abuse evidence; //foo-style empty path segments are an instant telltale. - Visits can no longer open on an external exit URL; favicon fetches skip hidden clients' referers/exits. Legacy analytics.json files are set aside as .bak-legacy on startup.
This commit is contained in:
@@ -434,7 +434,9 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
|
||||
|
||||
/**
|
||||
* Group abuse hits by IP and format each group as a row with the full paths
|
||||
* probed. Identical paths are collapsed into one entry with their hit count.
|
||||
* probed. Identical requests (same path and status class) are collapsed
|
||||
* into one entry with their hit count; a path's 404 probes and its real
|
||||
* (200) reads never merge.
|
||||
* The paths split into two lists: ``paths`` holds the 404 probes (flagged
|
||||
* paths — the ones that triggered abuse classification — first, then other
|
||||
* 404s) shown verbatim, query string included, and ``articles`` holds the
|
||||
@@ -465,7 +467,11 @@ export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) {
|
||||
g.lastClient = a.client
|
||||
}
|
||||
const path = a.path || ''
|
||||
const existing = g.pathCounts.get(path) || {
|
||||
// Collapse identical requests, but never merge a path's 404 probes with
|
||||
// its real (200) reads — a page probed while missing and later created
|
||||
// must show up in both columns, not flip to "articles read".
|
||||
const key = `${a.is_404 ? '4' : '2'}${path}`
|
||||
const existing = g.pathCounts.get(key) || {
|
||||
path,
|
||||
count: 0,
|
||||
firstStart: start,
|
||||
@@ -475,8 +481,7 @@ export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) {
|
||||
existing.count += 1
|
||||
if (start < existing.firstStart) existing.firstStart = start
|
||||
if (a.flag) existing.flag = true
|
||||
if (!a.is_404) existing.is_404 = false
|
||||
g.pathCounts.set(path, existing)
|
||||
g.pathCounts.set(key, existing)
|
||||
g.clientHashes.add(a.client)
|
||||
groups.set(ip, g)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user