LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reclassify sub-5s visits as crawlers, show crawler referers and abuser articles.

Browse Source 
Visits with under 5 s of total reported reading time are JS-running bots:
display() converts them to crawler hits (one per trail page, with referer
and UTM query) and excludes them from every aggregate.  Crawler referers
join the favicon fetch origins and render with their icon in the crawler
table.  Abuse hits now record the real response status, so the abuse table
splits 404 probes from the articles the abuser actually read (200 GETs),
shown as trail links like the visitor/crawler tables.
This commit is contained in:
LeoVasanko
2026-09-03 15:43:43 +00:00
parent 7631c9f0a3
commit 85ef968cd6
4 changed files with 123 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/analytics.md
+30 -12
View File
@@ -47,6 +47,9 @@ WebSocket reports actual navigations and active time spent on a page.
(anything calling
itself a "bot", plus known exceptions such as GoogleOther) are ignored
server-side, and their document GETs land in the crawler list instead.
Real-browser bots whose UA does not match still register a visit, but
their reported reading time stays under 5 seconds, so they are
reclassified as crawler hits at display time (see **Crawler hits** below).
No source-IP verification is done: a spoofed bot UA merely lands in the
crawler stats, and scanners that probe telltale paths are caught by the
abuse rules regardless. Reloads are not
@@ -90,7 +93,7 @@ WebSocket reports actual navigations and active time spent on a page.
("/" or `[a-z0-9_-]` segments), external ones are re-derived to the
https origin and accepted only when the client sent exactly that.
- **External-site favicons**: for every external https origin seen as a visit
referer or an exit link, the server fetches `{origin}/favicon.ico` in a
referer, a crawler-hit referer or an exit link, the server fetches `{origin}/favicon.ico` in a
background task (httpx, 8 s timeout, ≤ 64 KB, image content-types only —
SVG is sniffed from the body when served without an image type) and stores
the icon content-hashed on disk in the FileStore (served at `/_f/{name}`,
@@ -134,11 +137,19 @@ WebSocket reports actual navigations and active time spent on a page.
from the same client arrives within 10 seconds the hit is discarded;
otherwise it is written to `crawlers` — unless the client is hidden
(admin), in which case the hit is discarded on expiry too. Crawlers do not count as
visits or views. The `Accept-Language` header is stored on the shared
visits or views. Bots running real browsers can still slip past the UA
check: a visit whose total reported reading time stays under 5 seconds
(`_MIN_VISIT_READ`; durations are client-provided and trusted — such bots
report 0–2 s) is reclassified as crawler hits at display time, one hit
per internal trail page, and counts in no visit aggregate. The
`Accept-Language` header is stored on the shared
`Client` immediately; reverse-DNS host names and DB-IP geoip
country/city are filled in asynchronously, just like for real visits. In
the analytics viewer, crawler hits are grouped by client hash and shown as
a trail of internal pages that crawler visited; the crawler table lists
a trail of internal pages that crawler visited, preceded by its referer
when there is one — spiders often advertise their own site as the
referer, and it is rendered with its favicon like visit referers (crawler
referers are included in the favicon fetch origins). The crawler table lists
the most recent crawler first, with the most active as a tie-breaker.
- **Abuse (scanner) hits**: a 404 for a telltale path — any URL segment
starting with a dot (`/.env`, `/.git/config`) or ending in `.php` —
@@ -152,9 +163,12 @@ WebSocket reports actual navigations and active time spent on a page.
is persisted in the JSON file; the plain-404 counters are RAM-only. In the
viewer, abuse hits are grouped by IP (never by client/UA — scanners
randomize theirs) in a separate "Abuse" table. Identical paths are
collapsed into one entry with their hit count; flagged paths that
triggered classification are lifted to the top, followed by other 404s and
then document GETs from the abuser. Raw User-Agent strings are shown one
collapsed into one entry with their hit count. The 404 probes ("paths
abused": flagged paths that triggered classification first, then other
404s) are kept in a separate column from the real articles the abuser
actually read ("articles read": document GETs that returned 200, not the
404 fallback rendering — rendered as trail links like the visitor and
crawler tables, with the query string stripped). Raw User-Agent strings are shown one
per line with their occurrence counts, and the full lists are click-to-copy.
## Visits and sessions
@@ -219,14 +233,16 @@ Each `AbuseHit` record:
- `client` — 6-byte blake3 hash referencing `Analytics.clients`,
- `flag` — true for the path that triggered abuse classification (telltale
path or the 404 that crossed the threshold),
- `is_404` — true for 404 responses, false for document GETs from the
abuser.
- `is_404` — true for 404 responses (probed paths and 404-fallback document
GETs), false for real (200) document GETs — articles the abuser read.
Crawler hits are grouped by client hash in the analytics viewer; abuse hits
are grouped by IP alone (resolved from the referenced `Client`). In the
Abuse table identical paths are collapsed with their counts; flagged paths
that triggered classification are lifted to the top, followed by other 404s
and then document GETs from the abuser. Within each category paths are
Abuse table identical paths are collapsed with their counts, split into the
404 probes (flagged paths that triggered classification first, then other
404s, shown verbatim) and the 200 document GETs shown as trail links in the
separate articles column.
Within each list paths are
sorted by count descending, then by their earliest hit.
In the visitor and crawler tables, internal paths that returned a 404 status
@@ -237,7 +253,9 @@ to tell misses from real pages at a glance.
Aggregates are **not stored**; they are computed at display time by
`Store.display()` from the visit records (entry + `navs` log), skipping
hidden clients' visits. This is what allows a client to become hidden after
hidden clients' visits and short visits reclassified as crawler hits
(under `_MIN_VISIT_READ` seconds of total reported reading time). This is
what allows a client to become hidden after
navigations were already logged: no counts need reversing. The computed
shapes, part of the WebSocket payload (`Display` struct alongside `visits`,
`crawlers`, `abuse` and `clients`):
frontend/src/AnalyticsView.vue
+8 -1
View File
@@ -163,7 +163,7 @@ const favicons = computed(() => data.value?.favicons || {})
const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value))
const crawlers = computed(() => rangeData.value?.crawlers || [])
const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value))
const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, now.value))
const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, pageTree.value, now.value))
</script>
@@ -244,6 +244,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
<tbody>
<tr v-for="(c, i) in crawlerRows" :key="i">
<td class="trail">
<TrailLink v-if="c.refererStep" :step="c.refererStep" :favicons="favicons" @close="$emit('close')" />
<TrailLink v-for="(s, si) in c.pages" :key="si" :step="s" :count="s.count" @close="$emit('close')" />
</td>
<VisitorCell
@@ -270,6 +271,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
<thead>
<tr>
<th>paths abused</th>
<th>articles read</th>
<th>visitor</th>
<th class="last-seen">last seen</th>
</tr>
@@ -286,6 +288,11 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
<small v-if="a.paths.length > ABUSE_MAX_LINES" class="muted">+{{ a.paths.length - ABUSE_MAX_LINES }} more</small>
</div>
</td>
<td class="trail clickable-list"
@click="copyList(a.allArticles, $event)">
<TrailLink v-for="(s, si) in a.articles" :key="si" :step="s" :count="s.count" @close="$emit('close')" />
<small v-if="!a.articles.length" class="muted">—</small>
</td>
<VisitorCell
:ip="a.ip"
:ip-display="a.ipDisplay"
frontend/src/analytics/format.js
+31 -17
View File
@@ -370,7 +370,9 @@ export function mainDomain(host, limit = 24) {
/**
* Group raw crawler hits by client hash and format each group as a row showing
* every internal page that crawler visited. Rows are sorted by most recent hit
* first, with total hits as a tie-breaker.
* first, with total hits as a tie-breaker. The group's ``refererStep`` is the
* latest external referer seen for the crawler — spiders often advertise
* their own site there — rendered with its favicon like visit referers.
* ``clients`` maps client hashes to client records.
*/
export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) {
@@ -382,10 +384,12 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
clientHash: c.client,
client,
lastStart: 0,
referer: '',
pages: new Map(),
}
const start = new Date(c.start).getTime()
if (start > g.lastStart) g.lastStart = start
if (c.referer) g.referer = c.referer
if (c.entry?.startsWith('/')) {
const existing = g.pages.get(c.entry) || { count: 0, status: c.status || 200 }
existing.count += 1
@@ -410,6 +414,7 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
lastSeen: formatWhen(g.lastStart, now),
lastSeenIso: formatWhenIso(g.lastStart),
lastSeenLocal: formatWhenLocal(g.lastStart),
refererStep: stepOf(g.referer, titles),
pages: [...g.pages.entries()]
.sort((a, b) => b[1].count - a[1].count)
.map(([path, info]) => ({ ...stepOf(path, titles), count: info.count, status: info.status })),
@@ -430,16 +435,19 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
/**
* Group abuse hits by IP and format each group as a row with the full paths
* probed. Identical paths are collapsed into one entry with their hit count.
* Flagged paths (the ones that triggered abuse classification) are lifted to
* the top, followed by other 404s, then document GETs from the abuser. Within
* each category paths are sorted by count descending, then earliest first.
* The paths split into two lists: ``paths`` holds the 404 probes (flagged
* paths — the ones that triggered abuse classification — first, then other
* 404s) shown verbatim, query string included, and ``articles`` holds the
* real (200) document GETs as trail steps resolved against the page tree
* (query string stripped), rendered like the visitor/crawler trails. Within
* each list paths are sorted by count descending, then earliest first.
* Rows are sorted by most recent hit first. Visitor metadata comes from the
* latest client hash seen for the IP; ``clientCount`` tells the visitor cell
* how many distinct client variations the IP produced. Paths are shown
* verbatim (query string included), not resolved against the page tree.
* how many distinct client variations the IP produced.
* ``clients`` maps client hashes to client records.
*/
export function formatAbuseRows(abuse, clients, now = Date.now()) {
export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) {
const titles = buildTitleMap(pageTree)
const groups = new Map()
for (const a of abuse || []) {
const client = (clients || {})[a.client] || {}
@@ -481,13 +489,14 @@ export function formatAbuseRows(abuse, clients, now = Date.now()) {
.sort((a, b) => b.lastStart - a.lastStart)
.slice(0, 10)
.map((g) => {
const pathCategory = (p) => (p.flag ? 0 : p.is_404 ? 1 : 2)
const paths = [...g.pathCounts.values()].sort(
(a, b) =>
pathCategory(a) - pathCategory(b) ||
b.count - a.count ||
a.firstStart - b.firstStart,
)
const all = [...g.pathCounts.values()]
const byCount = (a, b) => b.count - a.count || a.firstStart - b.firstStart
const paths = all
.filter((p) => p.flag || p.is_404)
.sort((a, b) => (a.flag ? 0 : 1) - (b.flag ? 0 : 1) || byCount(a, b))
const articles = all.filter((p) => !p.flag && !p.is_404).sort(byCount)
const pathList = (list) =>
list.map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path)).join('\n')
const client = (clients || {})[g.lastClient] || {}
const host = client.host || ''
const isHost = !!host
@@ -501,9 +510,14 @@ export function formatAbuseRows(abuse, clients, now = Date.now()) {
flag: p.flag,
is_404: p.is_404,
})),
allPaths: paths
.map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path))
.join('\n'),
allPaths: pathList(paths),
articles: articles
.map((p) => {
const step = stepOf(p.path.split('?')[0], titles)
return step ? { ...step, count: p.count } : null
})
.filter(Boolean),
allArticles: pathList(articles),
clientCount: g.clientHashes.size,
ip: client.ip || g.ip,
ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip || g.ip) || client.ip || g.ip || '—',
pagerite/analytics.py
+54 -9
View File
@@ -14,7 +14,11 @@ that only fetch documents end up in the crawler list). JS-running crawlers
(Googlebot, GoogleOther, Applebot, ...) do connect and send messages, but
their UA gives them
away (``_is_bot_ua``) and their messages are ignored, so they land in the
crawler list too. Idle-time link preloads from pagerite.js carry an
crawler list too. Bots whose UA does not match are caught by engagement:
a visit whose total reported reading time is under ``_MIN_VISIT_READ``
seconds is reclassified as crawler hits at display time (durations are
client-provided and trusted — real-browser bots report 0–2 s), so it never
counts in the visit aggregates either. Idle-time link preloads from pagerite.js carry an
``x-pagerite-preload`` header and are not tracked at all — the navigation
message sent when the user actually navigates does the counting.
Admin clients send ``hide``: the client record is flagged ``hide``,
@@ -43,7 +47,7 @@ from collections.abc import Callable
from contextlib import suppress
from datetime import UTC, datetime, timedelta
from pathlib import Path
from urllib.parse import parse_qs, urlparse
from urllib.parse import parse_qs, urlencode, urlparse
import blake3
import msgspec
@@ -200,8 +204,10 @@ class AbuseHit(msgspec.Struct, omit_defaults=True):
Unlike crawler hits the full request path (query string included) is
kept: the interesting part is exactly which paths were probed.
``flag`` marks the path that triggered classification; ``is_404``
distinguishes 404 responses from document GETs made by the abuser.
Client metadata is held in ``Analytics.clients`` keyed by ``client``.
distinguishes 404 responses (probed paths and 404-fallback document
GETs) from real 200 document GETs — the abuser actually reading
articles. Client metadata is held in ``Analytics.clients`` keyed by
``client``.
"""
start: datetime
@@ -212,7 +218,7 @@ class AbuseHit(msgspec.Struct, omit_defaults=True):
#: True when this path triggered abuse classification (telltale path
#: or the 404 that crossed the threshold).
flag: bool = False
#: True for 404 responses; false for document GETs from the abuser.
#: True for 404 responses; false for real (200) document GETs.
is_404: bool = False
@@ -347,6 +353,11 @@ def _utm_tags(query: str) -> dict[str, str]:
_CRAWLER_TIMEOUT = timedelta(seconds=10)
#: Minimum total reported reading time (seconds, summed over the trail) for
#: a session to count as a visit; shorter sessions are JS-running bots and
#: are shown as crawler hits instead.
_MIN_VISIT_READ = 5
#: How long a failed favicon fetch suppresses retries for the same origin.
_FAVICON_RETRY = timedelta(days=7)
@@ -505,16 +516,43 @@ class Store:
def display(self) -> Display:
"""Build the viewer payload, excluding hidden clients.
Visits whose total reported reading time is under
``_MIN_VISIT_READ`` seconds are JS-running bots, not readers: they
are converted to crawler hits (one per internal trail page) and left
out of the visit list and every aggregate.
The aggregates (site visits, page views, transitions) are computed
here from the visit records rather than stored, so a client that
becomes hidden after navigations were already logged disappears
from every statistic. Internal-path navigations count as page
views; external https targets are transitions only.
"""
visits = [v for v in self.data.visits if not self._hidden(v.client)]
visits: list[Visit] = []
crawlers = [h for h in self.data.crawlers if not self._hidden(h.client)]
for visit in self.data.visits:
if self._hidden(visit.client):
continue
if sum(item.read for item in visit.trail.values()) >= _MIN_VISIT_READ:
visits.append(visit)
continue
query = urlencode(visit.utm)
first = True
for t, item in visit.trail.items():
if not item.to.startswith("/"):
continue
crawlers.append(
CrawlerHit(
start=t,
entry=item.to,
client=visit.client,
referer=visit.referer if first else "",
query=query if first else "",
status=item.status,
)
)
first = False
display = Display(
visits=visits,
crawlers=[h for h in self.data.crawlers if not self._hidden(h.client)],
crawlers=crawlers,
abuse=[h for h in self.data.abuse if not self._hidden(h.client)],
clients={h: c for h, c in self.data.clients.items() if not c.hide},
favicons={
@@ -601,7 +639,9 @@ class Store:
def favicon_origins_needed(self) -> list[str]:
"""External https origins seen in visits whose favicon needs fetching.
Covers visit referers and external exit targets (trail and navs).
Covers visit referers and external exit targets (trail and navs),
plus crawler-hit referers — spiders often advertise their own site
as the referer, so the icon identifies them in the crawler table.
Origins with a stored icon, or a miss younger than
``_FAVICON_RETRY``, are skipped.
"""
@@ -613,6 +653,9 @@ class Store:
origin = _origin(target.to)
if origin is not None:
origins.add(origin)
for hit in self.data.crawlers:
if hit.referer:
origins.add(hit.referer)
now = datetime.now(UTC)
return [
origin
@@ -673,6 +716,7 @@ class Store:
h.client,
h.entry + (f"?{h.query}" if h.query else ""),
start=h.start,
is_404=h.status != 200,
)
pending = [
h for h in self.pending_crawlers if self._client_ip(h.client) == ip
@@ -686,6 +730,7 @@ class Store:
h.client,
h.entry + (f"?{h.query}" if h.query else ""),
start=h.start,
is_404=h.status != 200,
)
self._abuse_hit(client_hash, path, flag=flag, is_404=is_404)
self._save()
@@ -779,7 +824,7 @@ class Store:
client_hash = self._ensure_client(ip, ua, lang, country=country)
if ip in self.data.abuse_ips:
flushed = self._flush_crawlers()
self._abuse_hit(client_hash, full_path, is_404=False, flag=False)
self._abuse_hit(client_hash, full_path, is_404=status != 200, flag=False)
self._save()
return flushed
now = datetime.now(UTC)