Fix analytics classification: ignore bot-UA pings, skip preload GETs.
JS-running crawlers (Googlebot, GoogleOther, Applebot) execute pagerite.js and send navigation pings, registering as visitors. Pings whose User-Agent matches _is_bot_ua (any "bot" token plus listed exceptions) are now ignored, so their document GETs flush to the crawler list as intended. No source verification: a spoofed bot UA merely lands in the crawler stats, and path-based abuse classification catches scanners regardless. Idle-time link preloads from pagerite.js were queued as pending crawler hits and flushed to the crawler list whenever the user navigated more than 10s later, so real visitors' subpage loads showed up as crawler hits. Preload fetches now carry an x-pagerite-preload header and the document GET handler skips tracking for them; the ping sent on actual navigation does the counting.
This commit is contained in:
@@ -337,7 +337,10 @@ import "overlayscrollbars/overlayscrollbars.css";
|
||||
}
|
||||
for (const url of urls) {
|
||||
if (pageCache.has(url)) continue;
|
||||
fetch(url)
|
||||
// x-pagerite-preload: idle cache warm-up, not a page view — the
|
||||
// server excludes these GETs from analytics (the ping sent on actual
|
||||
// navigation does the counting).
|
||||
fetch(url, { headers: { "x-pagerite-preload": "1" } })
|
||||
.then((r) => (r.ok && (r.headers.get("content-type") || "").includes("text/html")
|
||||
? r.text() : ""))
|
||||
.then((html) => { if (html) pageCache.set(url, html); })
|
||||
|
||||
Reference in New Issue
Block a user