Fix analytics classification: ignore bot-UA pings, skip preload GETs.

JS-running crawlers (Googlebot, GoogleOther, Applebot) execute pagerite.js
and send navigation pings, registering as visitors. Pings whose User-Agent
matches _is_bot_ua (any "bot" token plus listed exceptions) are now
ignored, so their document GETs flush to the crawler list as intended. No
source verification: a spoofed bot UA merely lands in the crawler stats,
and path-based abuse classification catches scanners regardless.

Idle-time link preloads from pagerite.js were queued as pending crawler
hits and flushed to the crawler list whenever the user navigated more than
10s later, so real visitors' subpage loads showed up as crawler hits.
Preload fetches now carry an x-pagerite-preload header and the document
GET handler skips tracking for them; the ping sent on actual navigation
does the counting.
This commit is contained in:
2026-08-22 18:23:38 +00:00
parent b7fc543a83
commit 6d2ae104d7
4 changed files with 56 additions and 8 deletions
+4 -1
View File
@@ -337,7 +337,10 @@ import "overlayscrollbars/overlayscrollbars.css";
}
for (const url of urls) {
if (pageCache.has(url)) continue;
fetch(url)
// x-pagerite-preload: idle cache warm-up, not a page view — the
// server excludes these GETs from analytics (the ping sent on actual
// navigation does the counting).
fetch(url, { headers: { "x-pagerite-preload": "1" } })
.then((r) => (r.ok && (r.headers.get("content-type") || "").includes("text/html")
? r.text() : ""))
.then((html) => { if (html) pageCache.set(url, html); })