Replace the pre-classified store (visits/crawlers/abuse lists written at
collection time, plus abuse_ips and in-memory pending/session tables) with
a raw append-only log: one Get record per document GET (full path, true
HTTP status, referer origin, preload flag) and one Msg per /_ws activity
message. Visitor/crawler/abuse classification, visit grouping (30-minute
inactivity gap), status/referer/UTM attribution and all aggregates are
derived in Store.display(), so future rule changes never invalidate stored
data. The viewer payload keeps its exact shape.
Fixes structurally:
- Abuser 404s on slug-format paths showed up as "articles read": the
not-found branch recorded the request twice through separate status
plumbing. Each request is now recorded once with its true status.
- 404 trail links never rendered red: cache-served navigations issue no
GET and the only real GET (the idle preload) was discarded before status
recording. Preloads are now recorded with pre=True, never counted, and
used for status attribution.
- formatAbuseRows merged a path's 404 probes and 200 reads into one entry;
the collapse is now keyed by (path, status class).
Rule improvements enabled by the redesign:
- The plain-404 abuse threshold counts within a 1-hour sliding window, so
long-time readers accumulating misses never classify (scanners spray).
- Hidden (admin) clients never trigger abuse classification: editing means
visiting not-found pages.
- /.well-known/ probes (RFC 8615, e.g. Chrome devtools) are never abuse
evidence; //foo-style empty path segments are an instant telltale.
- Visits can no longer open on an external exit URL; favicon fetches skip
hidden clients' referers/exits.
Legacy analytics.json files are set aside as .bak-legacy on startup.
LeoVasanko
eb2e8f8273