LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Collect analytics over a /_ws WebSocket instead of POST /_a pings

Browse Source
This commit is contained in:
LeoVasanko
2026-08-29 21:20:04 +00:00
parent 447a565b05
commit 31895065f8
4 changed files with 255 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/analytics.md
+46 -29
View File
@@ -9,9 +9,9 @@ directory, e.g. `localhost/analytics.json`).
`CrawlerHit`, `AbuseHit`, `Favicon`) and the `Store` (in-memory data + session map,
atomic JSON persistence).
- `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`),
the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics`
the `/_ws` activity WebSocket, and `WebSocket /_api/ws/analytics`
(admin-gated like every `/_api` endpoint).
- `frontend/src/pagerite.js` — client navigation pings and the 📊 pen.
- `frontend/src/pagerite.js` — the client activity channel and the 📊 pen.
- `frontend/src/AnalyticsView.vue` — viewer component rendered inside the
normal site layout on the `/_a` analytics page.
- `frontend/src/analytics-main.js` — page entry that mounts `AnalyticsView`
@@ -19,46 +19,61 @@ directory, e.g. `localhost/analytics.json`).
## What is collected
The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
`fr`, `to`, `hide` and `read` as query parameters (`fr` = source path;
falsy values are omitted):
The client (`pagerite.js`) keeps a WebSocket connection to `/_ws` for the
whole browsing session and sends activity messages over it — JSON text
frames matching the server's `Ping` msgspec struct with the fields `fr`
(source path), `to` (navigation target), `read` (active seconds on `fr`
since the last report) and `hide`; falsy fields are omitted. One channel
follows the session, so the activity of a visit stays tied together, and
while the user is active the accumulated reading time is flushed every few
seconds: the trail times are cumulative, so a disconnection simply leaves
the last reported time in place (no close beacon). After 5 minutes without
any activity the client closes the socket itself — a sleeping browser tab
would lose it anyway — and the next activity reconnects as a fresh session;
reconnects are attempted only on user activity, with an exponential backoff
between attempts so a failing endpoint is never hammered. Idle-time link preloads
stay plain `fetch()` calls so the browser may cache the responses; the
WebSocket reports actual navigations and active time spent on a page.
- **Initial page load**: only `to` — the loaded path — is sent, never `fr`
(an `fr` equal to `to` would log a bogus self-transition when a session
already exists, e.g. a second tab). This ping is what starts
already exists, e.g. a second tab). This message is what starts
the visit and counts the entry page view — the document GET alone records
nothing, so bots never register (admin browsing does register, but
flagged `hide`; see **Admins** below). JS-running crawlers
(Googlebot, GoogleOther, Applebot, ...) do ping, but their User-Agent
gives them away: pings whose UA matches `_is_bot_ua` (anything calling
(Googlebot, GoogleOther, Applebot, ...) do connect and report, but their
User-Agent gives them away: messages whose UA matches `_is_bot_ua`
(anything calling
itself a "bot", plus known exceptions such as GoogleOther) are ignored
server-side, and their document GETs land in the crawler list instead.
No source-IP verification is done: a spoofed bot UA merely lands in the
crawler stats, and scanners that probe telltale paths are caught by the
abuse rules regardless. Reloads are not
visits: the ping is skipped (PerformanceNavigationTiming `reload`), so a
visits: the message is skipped (PerformanceNavigationTiming `reload`), so a
refresh neither counts a second view nor logs a self-transition. The GET
handler stashes a cross-origin https `Referer` (origin part only —
unavailable to JS once the page has loaded) and any
`utm_*` query parameters in in-memory IP tables, consumed by the ping that
`utm_*` query parameters in in-memory IP tables, consumed by the first
message that
starts the visit; internal or absent referers never touch the referer table.
- **Internal fetch-navigations**: `to` is the target path, sent only after
the swap actually happened (a failed swap falls back to a full load,
whose initial ping counts the view instead — no gap, no double count).
whose initial message counts the view instead — no gap, no double count).
- **External links** (`https` only): `to` is the link's full URL. This is the
exit-link record; the user may continue navigating afterwards (new tab,
back), so the exit URL is not necessarily the last trail entry. Outbound
links are stored by full URL so several links to the same domain remain
distinct.
- **Excluded**: back/forward (popstate) navigations, navigating *to* the
analytics page (`/_a` — its GET is untracked, and the server rejects it
as a ping target anyway), and everything while the user has the editor
analytics page (`/_a` — its GET is untracked, and the server cannot
record it as a navigation target anyway), and everything while the user has
the editor
open (`body.editing`). Admin noise, not visits. Navigating *away* from
`/_a` does ping: the fetch-navigation already GET-ed the target page
without the preload header, and without the ping that GET would flush to
`/_a` does report: the fetch-navigation already GET-ed the target page
without the preload header, and without the message that GET would flush to
the crawler list.
- **Admins**: when SSO is in use and the session is known to be an admin,
the client still pings but adds `hide=1`. The activity is recorded as
the client still reports but adds `hide`. The activity is recorded as
usual (navigations and all), but the `hide` flag is set on the **client
record** — so it covers everything that client ever did: visits and
crawler hits from before the login included. Hidden clients never appear
@@ -81,7 +96,7 @@ falsy values are omitted):
extension matching the actual MIME). The origin → file name mapping is
recorded in `Analytics.favicons` (`Favicon.file`/`fetched`); misses are
recorded too and retried only after 7 days. Fetches are scheduled after
each ping and once at startup, which backfills icons for already-recorded
each activity message and once at startup, which backfills icons for already-recorded
data. The viewer payload carries `favicons` (origin → `/_f/...` path),
and the viewer shows the icon wherever an external site is mentioned:
referer/exit trail links in the visit table and the source/exit pills of
@@ -100,8 +115,8 @@ falsy values are omitted):
`host`; local/reserved/multicast addresses are skipped. If a DB-IP MMDB
file (`dbip-*.mmdb` or `dbip-*.mmdb.gz`) is present in the repository
root, it is loaded at startup and used to look up `country`/`city`. These
lookups run in background tasks after the event is stored, so the `/_a`
response is never delayed. The decompressed `dbip-*.mmdb` file is kept in
lookups run in background tasks after the event is stored, so WebSocket
message handling is never delayed. The decompressed `dbip-*.mmdb` file is kept in
the repository root and ignored by git. The CLI flag `--dbip`
(`uv run pagerite --dbip`) downloads the latest
`dbip-city-lite-YYYY-MM.mmdb.gz` from DB-IP before the server starts,
@@ -110,10 +125,11 @@ falsy values are omitted):
file is used.
- **Crawler hits**: every document GET is queued in RAM as a pending crawler
hit — except idle-time link preloads from pagerite.js, which carry an
`x-pagerite-preload` header and are not tracked at all (the ping sent when
the user actually navigates to a preloaded page does the counting; forging
`x-pagerite-preload` header and are not tracked at all (the navigation
message sent when the user actually navigates to a preloaded page does
the counting; forging
the header only hides a GET from the crawler stats, the path-based abuse
classification is unaffected). If a ping
classification is unaffected). If a message
from the same client arrives within 10 seconds the hit is discarded;
otherwise it is written to `crawlers` — unless the client is hidden
(admin), in which case the hit is discarded on expiry too. Crawlers do not count as
@@ -131,7 +147,7 @@ falsy values are omitted):
random-UA scanner no longer pollutes the crawler stats of the legitimate
bot it impersonates. Once classified, every document GET and 404 from the
IP is recorded as an abuse hit with the full request path (query string
included), and its pings are ignored. The classified IP set (`abuse_ips`)
included), and its activity messages are ignored. The classified IP set (`abuse_ips`)
is persisted in the JSON file; the plain-404 counters are RAM-only. In the
viewer, abuse hits are grouped by IP (never by client/UA — scanners
randomize theirs) in a separate "Abuse" table. Identical paths are
@@ -145,9 +161,9 @@ falsy values are omitted):
There are no cookies. A visit is tied together by a client hash — the first
6 bytes of a blake3 digest over the prettified IP (IPv4 unchanged, IPv6
/64 network), the raw `User-Agent` string and the extracted
`Accept-Language` tag. The first ping from a client hash starts a new
visit; subsequent pings extend it. Pings arriving with no known session
(server restart) start a fresh visit from the first ping — treated as
`Accept-Language` tag. The first message from a client hash starts a new
visit; subsequent messages extend it. Messages arriving with no known session
(server restart) start a fresh visit from the first message — treated as
missing data rather than dropped. The client-hash → visit map and the IP →
entry-referer/UTM tables are in-memory only; client metadata is stored in
`Analytics.clients` keyed by the client hash.
@@ -164,7 +180,7 @@ Each `Client` record:
- `ua` — raw `User-Agent` string,
- `ua_pretty` — compact display form of the UA (browser/OS/device) when
parsable, otherwise the raw string,
- `hide` — true for admin clients (`hide=1` ping): all their visits,
- `hide` — true for admin clients (`hide` message field): all their visits,
crawler hits and abuse hits are recorded but excluded from every
statistic and from the viewer payload.
@@ -180,7 +196,7 @@ Each `Visit` record:
reading time in seconds (`read`) and the most recent HTTP status seen
for the target (`status`). Re-visiting an already seen target updates
its item instead of appending.
- `navs` — every navigation ping (`fr`, `to`), keyed by its timestamp,
- `navs` — every navigation message (`fr`, `to`), keyed by its timestamp,
repeats included. The aggregates are computed from this log at display
time.
- `utm` — `utm_*` query parameters from the landing URL, as a dict.
@@ -227,7 +243,8 @@ shapes, part of the WebSocket payload (`Display` struct alongside `visits`,
- `transitions`: time series of page transitions, sparse nested dict
`from -> to -> bucket -> count` with 5-minute bucketing. `from` is the
referer origin or `"(direct)"` for initial loads, a page path for pings.
referer origin or `"(direct)"` for initial loads, a page path for
navigations.
- `views`: time series of page loads, `path -> bucket -> count`, sparse: only
non-zero 5-minute buckets exist (bucket key is its floored ISO timestamp).
Every load counts, including repeats within a visit; external exit origins
frontend/src/pagerite.js
+111 -54
View File
@@ -347,8 +347,8 @@ import "overlayscrollbars/overlayscrollbars.css";
for (const url of urls) {
if (pageCache.has(url)) continue;
// x-pagerite-preload: idle cache warm-up, not a page view — the
// server excludes these GETs from analytics (the ping sent on actual
// navigation does the counting).
// server excludes these GETs from analytics (the navigation message
// sent on actual navigation does the counting).
fetch(url, { headers: { "x-pagerite-preload": "1" } })
.then((r) => (r.ok && (r.headers.get("content-type") || "").includes("text/html")
? r.text() : ""))
@@ -432,61 +432,111 @@ import "overlayscrollbars/overlayscrollbars.css";
});
}, { passive: true });
// --- Analytics pings ---------------------------------------------------
// Fire-and-forget POSTs to /_a with the fields as query parameters (a
// beacon can carry no body, and query args show in server logs next to
// the document GET they refer to): on the initial page load (starts the
// visit — the server counts nothing from the document GET alone), for
// internal fetch-navigations, for external https exits, and on window
// close. ``read`` is the active time (ms) spent on ``fr``.
// Reading time pauses after 1 minute of inactivity and resumes on the
// next mouse/touch/scroll/keyboard event.
// Excluded: back/forward (popstate never pings), everything while the
// --- Analytics over WebSocket ------------------------------------------
// One /_ws connection follows the whole browsing session: the initial page
// load (starts the visit — the server counts nothing from the document GET
// alone), internal fetch-navigations, external https exits, and frequent
// active reading-time updates. Messages are JSON text frames matching the
// server's msgspec Ping struct: {fr?, to?, read?, hide?} — falsy fields
// are omitted. ``read`` is the active time (ms) accumulated on ``fr`` since
// the last report; reading time pauses after 1 minute of inactivity and
// resumes on the next mouse/touch/scroll/keyboard event. While the user is
// active, accumulated reading time is flushed every few seconds, so a
// disconnect simply leaves the last reported time on the server — no close
// beacon is needed. After 5 minutes without any activity the client closes
// the channel itself (a sleeping tab would lose it anyway); the next
// activity reconnects and the server sees a new session.
// Excluded: back/forward (popstate never reports), everything while the
// editor is open (body.editing — admin noise, not visits), and
// navigations TO the analytics page (/_a — admin machinery, and the
// server rejects it as a ping target anyway). Navigations AWAY from /_a
// must ping: load() already fetched the target page without the preload
// header, and without the ping that GET would flush to the crawler list.
// navigations TO the analytics page (/_a — admin machinery). Navigations
// AWAY from /_a must report: load() already fetched the target page
// without the preload header, and without the message that GET would flush
// to the crawler list.
// Admins (when SSO is actually in use — with no auth proxy "admin" is
// everyone's state) ping normally but with hide=1: the server then
// records nothing and scrubs any session the same browser accumulated
// before logging in, so admins never show up as visits or crawlers.
// everyone's state) report normally but with hide: the server then flags
// the client record, scrubbing everything it ever did from the statistics,
// so admins never show up as visits or crawlers.
// See docs/analytics.md.
// fetch wrapper: every key of ``params`` becomes a query arg on /_a
// (falsy values are omitted). Admins get hide=1. ``beacon`` uses
// sendBeacon when available, for unload-time pings.
function pingFetch(params, { beacon = false } = {}) {
const query = new URLSearchParams();
if (ssoAvailable && isAdmin) params = { ...params, hide: 1 };
for (const [key, value] of Object.entries(params)) {
if (value) query.set(key, value);
}
const url = `/_a?${query}`;
// The activity WebSocket. Messages sent before the connection opens are
// queued (the queue keeps the interim activity). Reconnects are driven by
// user activity only — never by timers while the page sits idle — with an
// exponential falloff between attempts so a failing endpoint cannot make
// us hammer the server (or trip its security limits). After a longer
// stretch without any activity we close the socket proactively: the user
// has moved on and left the tab open (a sleeping browser tab would lose
// the connection anyway), so the next activity reconnects and registers
// as a fresh session. Analytics must never break navigation: every send
// is wrapped, and a server without the endpoint just leaves the socket
// failing in the background.
let ws = null;
const wsQueue = [];
let wsReconnectMs = 1000;
let wsNotBefore = 0;
function activityWs() {
if (ws || Date.now() < wsNotBefore) return;
const url = new URL("/_ws", location.href);
url.protocol = url.protocol === "https:" ? "wss:" : "ws:";
try {
if (beacon && navigator.sendBeacon) {
navigator.sendBeacon(url);
} else {
fetch(url, { method: "POST", keepalive: true });
}
} catch { /* analytics must never break navigation */ }
ws = new WebSocket(url);
} catch {
return;
}
ws.onopen = () => {
wsReconnectMs = 1000;
for (const msg of wsQueue.splice(0)) ws.send(JSON.stringify(msg));
};
ws.onclose = () => {
ws = null;
// No timer here: the next user activity retries, after the backoff.
wsNotBefore = Date.now() + wsReconnectMs;
wsReconnectMs = Math.min(wsReconnectMs * 2, 30_000);
};
ws.onerror = () => ws.close();
}
function ping({ to, fr = currentPath, read = 0, beacon = false } = {}) {
function report(msg) {
if (document.body.classList.contains("editing")) return;
if (to === "/_a") return;
pingFetch({ fr, to, read: Math.round(read / 1000) }, { beacon });
if (msg.to === "/_a") return;
if (ssoAvailable && isAdmin) msg.hide = true;
activityWs();
if (ws?.readyState === WebSocket.OPEN) {
try {
ws.send(JSON.stringify(msg));
return;
} catch { /* fall through to queueing */ }
}
wsQueue.push(msg);
}
function ping({ to, fr = currentPath, read = 0 } = {}) {
// Reading-time updates from the analytics page itself are not tracked
// (/_a is admin machinery; the server would reject the path anyway).
if (!to && currentPath === "/_a") return;
const msg = {};
if (fr) msg.fr = fr;
if (to) msg.to = to;
const secs = Math.round(read / 1000);
if (secs > 0) msg.read = secs;
if (!msg.to && !msg.read) return;
report(msg);
}
// Active reading time for the current page. The clock stops after 1 minute
// without activity and restarts on the next mouse/touch/scroll/keyboard
// event.
// event. Every READ_FLUSH_MS of accumulated activity is reported. After
// IDLE_MS with no activity at all, the remaining read time is flushed and
// the WebSocket is closed: the user has moved on, and the next activity
// reconnects as a new session.
const INACTIVE_MS = 60_000;
const READ_FLUSH_MS = 5_000;
const IDLE_MS = 5 * 60_000;
let readStart = performance.now();
let readElapsed = 0;
let reading = true;
let readInactivityTimer = null;
let closePingedFor = null;
let idleTimer = null;
function markReadActivity() {
if (!reading) {
@@ -500,6 +550,24 @@ import "overlayscrollbars/overlayscrollbars.css";
reading = false;
}
}, INACTIVE_MS);
// Any activity is a sign of life: (re)connect the channel if it was
// dropped or idle-closed (not while editing — admin noise), and push
// the idle disconnect forward.
if (!document.body.classList.contains("editing")) activityWs();
clearTimeout(idleTimer);
idleTimer = setTimeout(() => {
if (ws) {
// Flush what is left unsent, then hang up. report() would try to
// reconnect a dead socket, which is exactly what we avoid here.
const left = takeReadTime();
if (Math.round(left / 1000) > 0) ping({ read: left });
ws.close();
}
}, IDLE_MS);
// Frequently update the article read time on the server.
if (readElapsed + performance.now() - readStart >= READ_FLUSH_MS) {
ping({ read: takeReadTime() });
}
}
function takeReadTime() {
@@ -519,28 +587,19 @@ import "overlayscrollbars/overlayscrollbars.css";
clearTimeout(readInactivityTimer);
}
function sendClosePing() {
if (closePingedFor === currentPath) return;
closePingedFor = currentPath;
const read = takeReadTime();
if (Math.round(read / 1000) <= 0) return;
ping({ read, beacon: true });
}
for (const ev of ["mousemove", "mousedown", "touchstart", "touchmove", "scroll", "keydown"]) {
addEventListener(ev, markReadActivity, { passive: true });
}
addEventListener("pagehide", sendClosePing);
// The initial page load pings too — it is what starts the visit and
// The initial page load reports too — it is what starts the visit and
// counts the entry page view (the document GET alone records nothing).
// It carries only ``to``: the server attributes the entry to the referer
// it saw on the document GET (unavailable to JS once loaded), and an
// ``fr`` equal to ``to`` would log a bogus self-transition when a
// session already exists (e.g. a second tab).
// Sent once per load, after the auth probes so the admin gate applies;
// the pageshow re-probe must not ping again. Reloads are not visits:
// pinging them would double-count the view and log a self-transition.
// the pageshow re-probe must not report again. Reloads are not visits:
// reporting them would double-count the view and log a self-transition.
let entryPinged = false;
function pingEntryOnce() {
if (entryPinged) return;
@@ -770,7 +829,6 @@ import "overlayscrollbars/overlayscrollbars.css";
// External link: the browser navigates; record the full https URL so
// different links to the same domain stay distinct in analytics.
if (url.protocol === "https:") {
closePingedFor = currentPath;
ping({ to: url.href, read: takeReadTime() });
}
return;
@@ -794,7 +852,6 @@ import "overlayscrollbars/overlayscrollbars.css";
const from = currentPath;
load(url).then((ok) => {
if (!ok) return;
closePingedFor = null;
ping({ to: url.pathname, fr: from, read: takeReadTime() });
resetReadTime();
});
pagerite/analytics.py
+53 -26
View File
@@ -1,18 +1,23 @@
"""Server-side visit analytics (collection only; see docs/analytics.md).
Events come from navigation pings POSTed to /_a by pagerite.js: the first
ping on page load starts a visit, later pings extend it, and pings with no
known session start a fresh one (missing data, not dropped). The document
Events come from pagerite.js over the /_ws WebSocket (``Ping`` messages as
JSON text frames): the first navigation message on page load starts a
visit, later messages extend it, and messages with no known session start a
fresh one (missing data, not dropped). Active reading time is reported as
frequent ``read`` updates while the user is active; the times are
cumulative per trail item, so a disconnect simply leaves the last logged
time in place. The document
GET handler stashes the entry referer (external https origin) and any
utm_* query parameters in in-memory IP tables, consumed when the ping
starts the visit; nothing is counted without a ping (plain bots that only
fetch documents end up in the crawler list). JS-running crawlers
(Googlebot, GoogleOther, Applebot, ...) do ping, but their UA gives them
away (``_is_bot_ua``) and their pings are ignored, so they land in the
utm_* query parameters in in-memory IP tables, consumed when the first
message starts the visit; nothing is counted without a message (plain bots
that only fetch documents end up in the crawler list). JS-running crawlers
(Googlebot, GoogleOther, Applebot, ...) do connect and send messages, but
their UA gives them
away (``_is_bot_ua``) and their messages are ignored, so they land in the
crawler list too. Idle-time link preloads from pagerite.js carry an
``x-pagerite-preload`` header and are not tracked at all — the ping sent
when the user actually navigates does the counting.
Admin clients ping with ``hide=1``: the client record is flagged ``hide``,
``x-pagerite-preload`` header and are not tracked at all — the navigation
message sent when the user actually navigates does the counting.
Admin clients send ``hide``: the client record is flagged ``hide``,
which covers everything that client ever did — visits and crawler hits
from before the login included. Aggregates (site visits, page views,
transitions) are not stored; they are computed at display time from the
@@ -70,6 +75,27 @@ def _compact_user_agent(ua: str) -> str:
return " ".join(p for p in parts if p).strip()
class Ping(msgspec.Struct, omit_defaults=True):
"""One client message on the /_ws activity WebSocket.
Sent as a JSON text frame (msgspec-encoded, decoded to str for the
wire). ``to`` set: a navigation — internal page path or external https
exit URL. ``read`` alone (with ``fr``): an active reading-time update
for the page ``fr``; these arrive frequently while the user is active
and accumulate on the trail item. ``hide`` flags the client as an
admin: everything it ever did is excluded from the statistics.
"""
#: Path of the page the activity happened on ("" for the initial load).
fr: str = ""
#: Navigation target: internal path or external https exit URL.
to: str = ""
#: Active reading time (seconds) spent on ``fr`` since the last report.
read: int = 0
#: Admin client: record but hide everything from the statistics.
hide: bool = False
class Client(msgspec.Struct, omit_defaults=True):
"""Client metadata shared by visits, crawler hits and abuse hits.
@@ -151,7 +177,7 @@ class Visit(msgspec.Struct, omit_defaults=True):
class CrawlerHit(msgspec.Struct, omit_defaults=True):
"""A document GET that was never followed by an analytics ping.
"""A document GET that was never followed by an activity message.
Client metadata is held in ``Analytics.clients`` keyed by ``client``.
"""
@@ -392,19 +418,19 @@ class Store:
#: client hash -> index of the current visit in data.visits
self.sessions: dict[bytes, int] = {}
#: ip -> external https origin of the latest document GET carrying
#: one, stashed for the visit the client's initial ping starts.
#: one, stashed for the visit the client's initial message starts.
#: Internal or absent referers never touch the table.
self.pending_referers: dict[str, str] = {}
#: ip -> utm_* query parameters from the latest document GET that
#: carried any, stashed for the visit the client's initial ping starts.
#: carried any, stashed for the visit the client's initial message starts.
#: Only non-empty sets are stored, so a later parameter-less page
#: does not overwrite an earlier tagged landing URL.
self.pending_utms: dict[str, dict[str, str]] = {}
#: Document GETs that have not yet been matched by a ping. Kept
#: Document GETs that have not yet been matched by a message. Kept
#: in RAM only; expired entries are written to ``data.crawlers``.
self.pending_crawlers: list[CrawlerHit] = []
#: client hash -> {path: status} for recent document GETs, consumed
#: by the matching ping to record the status of each visited path.
#: by the matching message to record the status of each visited path.
self.pending_statuses: dict[bytes, dict[str, int]] = {}
#: ip -> number of plain (non-telltale) 404s seen, in RAM only;
#: reaching ``_ABUSE_404_THRESHOLD`` classifies the IP as abuse.
@@ -721,16 +747,16 @@ class Store:
) -> list[bytes]:
"""Stash the entry referer/UTM tags and queue a pending crawler hit.
Nothing is counted here — the client's initial /_a ping starts the
visit (only non-admin clients ping). Only a cross-origin https
Nothing is counted here — the client's first /_ws message starts the
visit (only non-admin clients report). Only a cross-origin https
referer updates the table; an internal or absent referer leaves any
stashed origin untouched. UTM parameters are kept only when the
landing URL actually carries them, so a subsequent parameter-less page
does not erase an earlier tagged landing.
Every document GET is also queued as a pending crawler hit. If a ping
from the same client arrives within ``_CRAWLER_TIMEOUT``, the hit is
discarded; otherwise it is flushed to ``data.crawlers``. The
Every document GET is also queued as a pending crawler hit. If a
message from the same client arrives within ``_CRAWLER_TIMEOUT``, the
hit is discarded; otherwise it is flushed to ``data.crawlers``. The
Accept-Language header is stored on the client record immediately;
host/geoip are filled in later by async enrichment.
@@ -794,15 +820,16 @@ class Store:
hide: bool = False,
read: int = 0,
) -> tuple[int | None, list[bytes]]:
"""Record a client navigation ping ({from, to, read} from pagerite.js).
"""Record a client activity message (``Ping`` from pagerite.js over /_ws).
``to`` is an internal path ("/...") or an https URL for exit links; a
missing/empty ``to`` means the page is being closed and only the
missing/empty ``to`` means a pure reading-time update and only the
``read`` time should be recorded. The transition is always counted when
``to`` is present; the trail only grows on first sight of a page within
the visit. ``read`` is the active time (seconds) spent on ``from_``.
the visit. ``read`` is the active time (seconds) spent on ``from_``
since the previous report.
A ping with no known session starts a fresh visit, consuming the
A message with no known session starts a fresh visit, consuming the
referer and UTM tags stashed by the document GET if there are any.
``hide`` is set by admin clients: the client record is flagged
@@ -811,7 +838,7 @@ class Store:
normally. Hidden clients are excluded from every statistic and list
at display time, and their pending crawler hits are discarded.
Pings from IPs classified as abuse, and pings whose User-Agent
Messages from IPs classified as abuse, and messages whose User-Agent
claims a JS-running crawler identity (``_is_bot_ua``), are ignored
entirely — the crawler's pending hits stay queued and flush to
``data.crawlers`` normally.
pagerite/app.py
+45 -38
View File
@@ -32,10 +32,10 @@ from xml.sax.saxutils import escape as xml_escape
import blake3
import httpx
import msgspec
from fastapi import (
FastAPI,
HTTPException,
Query,
Request,
WebSocket,
WebSocketDisconnect,
@@ -935,7 +935,7 @@ async def delete_page(path: str) -> None:
_SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9_-]*$")
def _client_ip(request: Request) -> str:
def _client_ip(request: Request | WebSocket) -> str:
"""Client IP: first X-Forwarded-For hop (we sit behind a proxy), else
the direct peer."""
forwarded = request.headers.get("x-forwarded-for", "").split(",")[0].strip()
@@ -1108,48 +1108,54 @@ async def analytics_page(request: Request) -> Response:
)
@app.post("/_a", status_code=204)
async def analytics_ping(
request: Request,
fr: str = Query(""),
to: str | None = Query(None),
hide: int = Query(0),
read: int = Query(0),
) -> None:
"""Record a navigation ping (?fr=&to=&hide=&read=); fire-and-forget.
@app.websocket("/_ws")
async def activity_ws(ws: WebSocket) -> None:
"""Collect visitor activity: navigations and reading-time updates.
The initial page-load ping carries only ``to``: the entry is attributed
to the referer/UTM tags stashed by the document GET (see _track_entry),
which JS cannot see once the page has loaded.
The reverse-DNS and DB-IP geoip lookups happen in a background task so
the response is never delayed by slow DNS or the first MMDB decompress.
Public, like the pages themselves (only /_api is gated); one connection
follows a browsing session. Messages are ``analytics.Ping`` structs as
JSON text frames; ``to`` set is a navigation, ``read`` alone a
reading-time update. The reverse-DNS and DB-IP geoip lookups happen in
background tasks so message handling is never delayed by slow DNS or
the first MMDB decompress.
"""
ip = _client_ip(request)
visit_index, flushed_clients = analytics_store.ping(
fr,
to,
ip,
request.headers.get("user-agent", ""),
request.headers.get("accept-language", ""),
hide=bool(hide),
read=read,
)
if visit_index is not None:
visit = analytics_store.data.visits[visit_index]
asyncio.create_task(_enrich_client(visit.client))
_schedule_client_enrichment(flushed_clients)
_schedule_favicon_fetch()
await ws.accept()
ip = _client_ip(ws)
ua = ws.headers.get("user-agent", "")
accept_language = ws.headers.get("accept-language", "")
try:
while True:
text = await ws.receive_text()
try:
msg = msgspec.json.decode(text.encode(), type=analytics.Ping)
except msgspec.DecodeError:
continue
visit_index, flushed_clients = analytics_store.ping(
msg.fr,
msg.to or None,
ip,
ua,
accept_language,
hide=msg.hide,
read=msg.read,
)
if visit_index is not None:
visit = analytics_store.data.visits[visit_index]
asyncio.create_task(_enrich_client(visit.client))
_schedule_client_enrichment(flushed_clients)
_schedule_favicon_fetch()
except WebSocketDisconnect:
pass
def _track_entry(path: str, request: Request, *, status: int = 200) -> list[bytes]:
"""Stash the referer/UTM tags and queue a pending crawler hit for the GET.
Nothing is counted on the GET itself — the client's /_a ping starts the
visit, so bots never register as visits (JS-running crawlers ping too,
but the ping handler ignores known bot UAs). (Admin clients ping too,
but with hide=1, which flags their visit hidden: it is recorded but
excluded from all statistics and from the crawler list.)
Nothing is counted on the GET itself — the client's first /_ws message
starts the visit, so bots never register as visits (JS-running crawlers
connect too, but the WebSocket handler ignores known bot UAs). (Admin
clients report too, but with hide, which flags their visit hidden: it is
recorded but excluded from all statistics and from the crawler list.)
The devserver's health probe (``GET /?from=devserver.py`` from
``127.0.0.1``) is ignored: it is not real traffic and would otherwise be
@@ -1161,7 +1167,8 @@ def _track_entry(path: str, request: Request, *, status: int = 200) -> list[byte
"""
if request.headers.get("x-pagerite-preload"):
# Idle-time page-cache warm-up by pagerite.js, not a page view: the
# ping sent when the user actually navigates does the counting.
# activity message sent when the user actually navigates does the
# counting.
# (Forging the header only hides a GET from the crawler stats; the
# path-based abuse classification is unaffected.)
return []