LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

analytics: record and show the rendered content language of page views

Browse Source 
GETs store the resolved page language, activity pings carry <html lang>
(including a new ping on in-place language switches, which also keeps the
switch's GET out of crawler classification), and trail items/crawler hits
surface it. The viewer shows flags discreetly: nothing on single-language
sites or primary-language-only rows, one leading flag for uniform visits,
transition flags on mixed trails, a flag row per crawler.
This commit is contained in:
LeoVasanko
2026-09-15 05:20:57 +00:00
parent 4cd8dbfc73
commit 3b1804b093
8 changed files with 286 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/analytics.md
+26 -7
View File
@@ -38,6 +38,9 @@ Each `Get` record (one per served document):
(`x-pagerite-preload` header): never counted as a view, crawler hit or (`x-pagerite-preload` header): never counted as a view, crawler hit or
abuse — recorded only so a navigation later served from the in-memory page abuse — recorded only so a navigation later served from the in-memory page
cache (which issues no GET at all) can be attributed this GET's status, cache (which issues no GET at all) can be attributed this GET's status,
- `lang` — rendered content language of the served document (the resolved
language of a localized page), `""` for non-localized responses (404
probes, reserved paths),
- `client` — 6-byte blake3 hash referencing `Analytics.clients`. - `client` — 6-byte blake3 hash referencing `Analytics.clients`.
304 revalidation responses return before recording and are not logged. 304 revalidation responses return before recording and are not logged.
@@ -51,7 +54,9 @@ Each `Msg` record (one per pagerite.js activity message over `/_ws`):
- `to` — navigation target (validated at record time: internal slug path or - `to` — navigation target (validated at record time: internal slug path or
external https URL; anything else is dropped — sanitation, not external https URL; anything else is dropped — sanitation, not
classification), classification),
- `read` — active seconds spent on `fr` since the previous report. - `read` — active seconds spent on `fr` since the previous report,
- `lang` — rendered language reported by the client for the page the
activity happened on (the page's `<html lang>`; `""` from old clients).
Each `Client` record (shared by every event, keyed by hash): Each `Client` record (shared by every event, keyed by hash):
@@ -104,7 +109,9 @@ The client (`pagerite.js`) keeps a WebSocket connection to `/_ws` for the
whole browsing session and sends activity messages over it — JSON text whole browsing session and sends activity messages over it — JSON text
frames matching the server's `Ping` msgspec struct with the fields `fr` frames matching the server's `Ping` msgspec struct with the fields `fr`
(source path), `to` (navigation target), `read` (active seconds on `fr` (source path), `to` (navigation target), `read` (active seconds on `fr`
since the last report) and `hide`; falsy fields are omitted. One channel since the last report), `lang` (the rendered language of the page the
activity happened on — its `<html lang>`) and `hide`; falsy fields are
omitted. One channel
follows the session, so the activity of a visit stays tied together, and follows the session, so the activity of a visit stays tied together, and
while the user is active the accumulated reading time is flushed every few while the user is active the accumulated reading time is flushed every few
seconds: the times are incremental, so a disconnection simply leaves the seconds: the times are incremental, so a disconnection simply leaves the
@@ -173,10 +180,16 @@ for misses.
(`_SESSION_GAP`). A fresh page load with an already-open visit (second (`_SESSION_GAP`). A fresh page load with an already-open visit (second
tab) extends it, logging a `(direct)` transition. The visit's trail holds tab) extends it, logging a `(direct)` transition. The visit's trail holds
first-seen targets in order; `read` updates accumulate active seconds on first-seen targets in order; `read` updates accumulate active seconds on
the trail item matching `fr`. Each trail item's HTTP status comes from the trail item matching `fr` (preferring the item whose language matches
the report, so seconds after a language switch land on the new-language
step). Each trail item's HTTP status comes from
the client's latest GET for that path — preloads included, which is what the client's latest GET for that path — preloads included, which is what
allows 404 pages to render red in the viewer even when the navigation allows 404 pages to render red in the viewer even when the navigation
itself was served from the page cache. The entry page's referer and itself was served from the page cache. Each trail item also carries the
rendered language: the client's report, for the entry page falling back
to its GET's rendered language (old clients don't send one); a page
re-visited in a different language becomes a distinct trail step instead
of merging into the existing item. The entry page's referer and
`utm_*` tags come from the GET that loaded it (within 10 s before the `utm_*` tags come from the GET that loaded it (within 10 s before the
first message). first message).
- **Crawler hits**: a document GET no activity message matched within - **Crawler hits**: a document GET no activity message matched within
@@ -232,6 +245,10 @@ to tell misses from real pages at a glance.
The `Display` payload contains the derived `visits`, `crawlers` and `abuse` The `Display` payload contains the derived `visits`, `crawlers` and `abuse`
rows (structs `Visit`/`Nav`/`TrailItem`, `CrawlerHit`, `AbuseHit` — display rows (structs `Visit`/`Nav`/`TrailItem`, `CrawlerHit`, `AbuseHit` — display
DTOs only, never persisted), the visible `clients`, the fetched `favicons`, DTOs only, never persisted), the visible `clients`, the fetched `favicons`,
the site language context (`multilingual` — translation languages are
configured, so the viewer can suppress language UI on single-language
sites — and `primary_lang` — the front page's primary language, so the
viewer can skip the primary-language default case),
and the aggregates below. and the aggregates below.
Each derived `Visit`: Each derived `Visit`:
@@ -243,8 +260,9 @@ Each derived `Visit`:
- `trail` — the entry page and everything seen afterwards, keyed by the - `trail` — the entry page and everything seen afterwards, keyed by the
timestamp of first sight (insertion order = first-seen order). Each item timestamp of first sight (insertion order = first-seen order). Each item
holds `to` (page path or external exit URL), the accumulated active holds `to` (page path or external exit URL), the accumulated active
reading time in seconds (`read`) and the most recent HTTP status seen reading time in seconds (`read`), the most recent HTTP status seen
for the target (`status`), for the target (`status`) and the rendered language (`lang`; a page
seen in two languages within one visit gets one item per language),
- `navs` — every navigation (`fr`, `to`), keyed by its timestamp, repeats - `navs` — every navigation (`fr`, `to`), keyed by its timestamp, repeats
included. The aggregates are computed from this log, included. The aggregates are computed from this log,
- `utm` — `utm_*` query parameters from the landing URL, as a dict. - `utm` — `utm_*` query parameters from the landing URL, as a dict.
@@ -257,7 +275,8 @@ Each derived `CrawlerHit`:
- `referer` — external https origin of the request, `""` for direct/none, - `referer` — external https origin of the request, `""` for direct/none,
- `query` — raw query string of the request, - `query` — raw query string of the request,
- `status` — HTTP status of the served response (200 for a real page, 404 - `status` — HTTP status of the served response (200 for a real page, 404
for a category placeholder or missing page). for a category placeholder or missing page),
- `lang` — rendered content language of the served document (from the GET).
Each derived `AbuseHit`: Each derived `AbuseHit`:
frontend/src/AnalyticsView.vue
+30 -3
View File
@@ -160,9 +160,15 @@ watch(range, (r) => {
const clients = computed(() => data.value?.clients || {}) const clients = computed(() => data.value?.clients || {})
const favicons = computed(() => data.value?.favicons || {}) const favicons = computed(() => data.value?.favicons || {})
const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value)) // Site language context from the payload: drives the discreet rendered-
// language markers in the visit/crawler rows (multilingual sites only).
const site = computed(() => ({
multilingual: !!data.value?.multilingual,
primaryLang: data.value?.primary_lang || '',
}))
const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value, site.value))
const crawlers = computed(() => rangeData.value?.crawlers || []) const crawlers = computed(() => rangeData.value?.crawlers || [])
const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value)) const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value, site.value))
const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, pageTree.value, now.value)) const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, pageTree.value, now.value))
</script> </script>
@@ -211,7 +217,8 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
<td class="trail"> <td class="trail">
<TrailLink v-if="v.refererStep" :step="v.refererStep" :favicons="favicons" @close="$emit('close')" /> <TrailLink v-if="v.refererStep" :step="v.refererStep" :favicons="favicons" @close="$emit('close')" />
<span v-if="v.utm && v.utm !== '—'" class="utm-tag small muted" :title="v.utmTitle">{{ v.utm }}</span> <span v-if="v.utm && v.utm !== '—'" class="utm-tag small muted" :title="v.utmTitle">{{ v.utm }}</span>
<TrailLink v-for="(s, si) in v.trail" :key="si" :step="s" :favicons="favicons" @close="$emit('close')" /> <span v-if="v.rowFlag" class="flag" v-html="v.rowFlag" :title="v.rowFlagTitle"></span>
<TrailLink v-for="(s, si) in v.trail" :key="si" :step="s" :favicons="favicons" :flags="s.langFlags" @close="$emit('close')" />
</td> </td>
<VisitorCell <VisitorCell
:ip="v.ip" :ip="v.ip"
@@ -248,6 +255,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
<td class="trail"> <td class="trail">
<TrailLink v-if="c.refererStep" :step="c.refererStep" :favicons="favicons" @close="$emit('close')" /> <TrailLink v-if="c.refererStep" :step="c.refererStep" :favicons="favicons" @close="$emit('close')" />
<TrailLink v-for="(s, si) in c.pages" :key="si" :step="s" :count="s.count" @close="$emit('close')" /> <TrailLink v-for="(s, si) in c.pages" :key="si" :step="s" :count="s.count" @close="$emit('close')" />
<span v-for="(f, fi) in c.readFlags" :key="fi" class="flag" v-html="f.flag" :title="f.name"></span>
</td> </td>
<VisitorCell <VisitorCell
:ip="c.ip" :ip="c.ip"
@@ -484,6 +492,25 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
vertical-align: bottom; vertical-align: bottom;
} }
/* Same flag chip as the visitor cells (VisitorCell.vue); the flags here
mark the language the page was read in. */
.visit-table .flag {
display: inline-flex;
width: 18px;
height: 12px;
border-radius: 2px;
overflow: hidden;
border: 1px solid var(--line);
box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
vertical-align: middle;
}
.visit-table .flag :deep(svg) {
width: 100%;
height: 100%;
display: block;
}
.visit-table .clickable-list { .visit-table .clickable-list {
cursor: pointer; cursor: pointer;
max-width: 22rem; max-width: 22rem;
frontend/src/TrailLink.vue
+21
View File
@@ -6,6 +6,7 @@ const props = defineProps({
step: { type: Object, required: true }, step: { type: Object, required: true },
count: { type: Number, default: 0 }, count: { type: Number, default: 0 },
favicons: { type: Object, default: null }, favicons: { type: Object, default: null },
flags: { type: Array, default: () => [] },
}) })
defineEmits(['close']) defineEmits(['close'])
@@ -38,6 +39,7 @@ const title = computed(() => {
<small v-if="count > 1" class="muted">{{ formatCount(count) }}×</small> <small v-if="count > 1" class="muted">{{ formatCount(count) }}×</small>
<img v-if="favicon" class="favicon" :src="favicon" alt="" /> <img v-if="favicon" class="favicon" :src="favicon" alt="" />
<span>{{ step.slug }}</span> <span>{{ step.slug }}</span>
<span v-for="(f, fi) in flags" :key="fi" class="flag" v-html="f"></span>
</a> </a>
</template> </template>
@@ -48,4 +50,23 @@ const title = computed(() => {
margin-right: 0.25em; margin-right: 0.25em;
vertical-align: -0.1em; vertical-align: -0.1em;
} }
/* Same flag chip as the visitor cells (VisitorCell.vue). */
.flag {
display: inline-flex;
width: 18px;
height: 12px;
margin-left: 0.25em;
border-radius: 2px;
overflow: hidden;
border: 1px solid var(--line);
box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
vertical-align: middle;
}
.flag :deep(svg) {
width: 100%;
height: 100%;
display: block;
}
</style> </style>
frontend/src/analytics/format.js
+60 -6
View File
@@ -2,6 +2,7 @@
* Formatters and aggregators for summary sections: totals and the recent * Formatters and aggregators for summary sections: totals and the recent
* visit trail. * visit trail.
*/ */
import { flagFor, langName } from '../langs.js'
/** /**
* IPv4 unchanged, IPv6 returns the /64 network prefix in compact form. * IPv4 unchanged, IPv6 returns the /64 network prefix in compact form.
@@ -373,7 +374,7 @@ export function mainDomain(host, limit = 24) {
* their own site there — rendered with its favicon like visit referers. * their own site there — rendered with its favicon like visit referers.
* ``clients`` maps client hashes to client records. * ``clients`` maps client hashes to client records.
*/ */
export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) { export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now(), site = { multilingual: false, primaryLang: '' }) {
const titles = buildTitleMap(pageTree) const titles = buildTitleMap(pageTree)
const groups = new Map() const groups = new Map()
for (const c of crawlers || []) { for (const c of crawlers || []) {
@@ -384,10 +385,12 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
lastStart: 0, lastStart: 0,
referer: '', referer: '',
pages: new Map(), pages: new Map(),
langs: new Set(),
} }
const start = new Date(c.start).getTime() const start = new Date(c.start).getTime()
if (start > g.lastStart) g.lastStart = start if (start > g.lastStart) g.lastStart = start
if (c.referer) g.referer = c.referer if (c.referer) g.referer = c.referer
if (c.lang) g.langs.add(c.lang)
if (c.entry?.startsWith('/')) { if (c.entry?.startsWith('/')) {
const existing = g.pages.get(c.entry) || { count: 0, status: c.status || 200 } const existing = g.pages.get(c.entry) || { count: 0, status: c.status || 200 }
existing.count += 1 existing.count += 1
@@ -408,6 +411,11 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
const client = g.client || {} const client = g.client || {}
const host = client.host || '' const host = client.host || ''
const isHost = !!host const isHost = !!host
// Rendered languages read, shown only when they say something the
// primary language alone would not (multilingual sites only).
const langs = [...g.langs].sort()
const showLangs =
site.multilingual && (langs.length > 1 || (langs[0] && langs[0] !== site.primaryLang))
return { return {
lastSeen: formatWhen(g.lastStart, now), lastSeen: formatWhen(g.lastStart, now),
lastSeenIso: formatWhenIso(g.lastStart), lastSeenIso: formatWhenIso(g.lastStart),
@@ -416,6 +424,9 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
pages: [...g.pages.entries()] pages: [...g.pages.entries()]
.sort((a, b) => b[1].count - a[1].count) .sort((a, b) => b[1].count - a[1].count)
.map(([path, info]) => ({ ...stepOf(path, titles), count: info.count, status: info.status })), .map(([path, info]) => ({ ...stepOf(path, titles), count: info.count, status: info.status })),
readFlags: showLangs
? langs.map((l) => ({ flag: flagFor(l), name: langName(l) })).filter((f) => f.flag)
: [],
ip: client.ip || '', ip: client.ip || '',
ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip) || client.ip || '—', ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip) || client.ip || '—',
isHost, isHost,
@@ -550,23 +561,44 @@ export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) {
* Format raw visit records as rows for a technical table. Returns objects * Format raw visit records as rows for a technical table. Returns objects
* with display strings; missing values become "—". ``trail`` starts with the * with display strings; missing values become "—". ``trail`` starts with the
* external referer (when present), then the entry page and any further internal * external referer (when present), then the entry page and any further internal
* pages or external exit origins. Only the 20 most recent visits are shown. * pages or external exit origins; consecutive views of the same page (e.g. a
* ``clients`` maps client hashes to client records. * language switch re-view) merge into one step that keeps the
* consecutive-distinct rendered languages, summed read time, and the latest
* status. On multilingual sites the rendered languages surface as flag
* icons: a visit read entirely in one non-primary language gets ``rowFlag``,
* and a visit spanning languages gets per-step ``langFlags`` markers where
* the language begins or changes. Only the 20 most recent visits are shown.
* ``clients`` maps client hashes to client records; ``site`` carries the
* payload's multilingual/primary-language context.
*/ */
export function formatVisitRows(visits, clients, pageTree, now = Date.now()) { export function formatVisitRows(visits, clients, pageTree, now = Date.now(), site = { multilingual: false, primaryLang: '' }) {
const titles = buildTitleMap(pageTree) const titles = buildTitleMap(pageTree)
return [...(visits || [])].reverse().slice(0, 20).map((v) => { return [...(visits || [])].reverse().slice(0, 20).map((v) => {
const client = (clients || {})[v.client] || {} const client = (clients || {})[v.client] || {}
const trail = Object.values(v.trail || {}) const steps = Object.values(v.trail || {})
.map((item) => { .map((item) => {
const step = stepOf(item.to, titles) const step = stepOf(item.to, titles)
if (step) { if (step) {
if (item.read) step.readSeconds = item.read if (item.read) step.readSeconds = item.read
if (item.status) step.status = item.status if (item.status) step.status = item.status
if (item.lang) step.lang = item.lang
} }
return step return step
}) })
.filter(Boolean) .filter(Boolean)
const trail = []
for (const step of steps) {
const prev = trail[trail.length - 1]
if (prev && prev.path === step.path) {
if (step.lang && step.lang !== prev.langs[prev.langs.length - 1]) prev.langs.push(step.lang)
if (step.readSeconds) prev.readSeconds = (prev.readSeconds || 0) + step.readSeconds
if (step.status) prev.status = step.status
} else {
step.langs = step.lang ? [step.lang] : []
trail.push(step)
}
}
const distinctLangs = new Set(trail.flatMap((s) => s.langs))
const utmKeys = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content'] const utmKeys = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content']
const utmValues = utmKeys.map((k) => (v.utm || {})[k]).filter(Boolean) const utmValues = utmKeys.map((k) => (v.utm || {})[k]).filter(Boolean)
const utm = utmValues.length ? utmValues.join(' · ') : '' const utm = utmValues.length ? utmValues.join(' · ') : ''
@@ -576,7 +608,7 @@ export function formatVisitRows(visits, clients, pageTree, now = Date.now()) {
const dash = (s) => (s || '—') const dash = (s) => (s || '—')
const host = client.host || '' const host = client.host || ''
const isHost = !!host const isHost = !!host
return { const row = {
lastSeen: formatWhen(v.start, now), lastSeen: formatWhen(v.start, now),
lastSeenIso: formatWhenIso(v.start), lastSeenIso: formatWhenIso(v.start),
lastSeenLocal: formatWhenLocal(v.start), lastSeenLocal: formatWhenLocal(v.start),
@@ -596,5 +628,27 @@ export function formatVisitRows(visits, clients, pageTree, now = Date.now()) {
utm: utm || '—', utm: utm || '—',
utmTitle, utmTitle,
} }
if (site.multilingual && distinctLangs.size) {
if (distinctLangs.size === 1) {
const [tag] = distinctLangs
const flag = flagFor(tag)
if (flag && tag !== site.primaryLang) {
row.rowFlag = flag
row.rowFlagTitle = langName(tag)
}
} else {
// Flag the steps where the rendered language begins or changes;
// lang-less steps keep the comparison chain going, they never flag.
let lastLang = null
for (const step of trail) {
if (!step.langs.length) continue
if (!lastLang || step.langs[step.langs.length - 1] !== lastLang) {
step.langFlags = step.langs.map(flagFor).filter(Boolean)
}
lastLang = step.langs[step.langs.length - 1]
}
}
}
return row
}) })
} }
frontend/src/pagerite.js
+6
View File
@@ -637,6 +637,8 @@ import { reconnectPolicy, socketSlot, watchConnecting } from "./reconnect";
if (to) msg.to = to; if (to) msg.to = to;
const secs = Math.round(read / 1000); const secs = Math.round(read / 1000);
if (secs > 0) msg.read = secs; if (secs > 0) msg.read = secs;
const lang = document.documentElement.lang;
if (lang) msg.lang = lang;
if (!msg.to && !msg.read) return; if (!msg.to && !msg.read) return;
report(msg); report(msg);
} }
@@ -815,6 +817,10 @@ import { reconnectPolicy, socketSlot, watchConnecting } from "./reconnect";
const y = scrollY; // a language switch is not a navigation: keep scroll const y = scrollY; // a language switch is not a navigation: keep scroll
await load(currentPath, false); await load(currentPath, false);
scrollTo(0, y); scrollTo(0, y);
// Log the switch as a trail event in the new language (load() updated
// <html lang>): the ping matches the switch's GET server-side, so it is
// not misclassified as a crawler hit.
ping({ to: currentPath });
}); });
// --- Fetch navigation ------------------------------------------------ // --- Fetch navigation ------------------------------------------------
pagerite/analytics.py
+114 -24
View File
@@ -2,8 +2,9 @@
Raw recording, display-time classification. Every document GET is appended Raw recording, display-time classification. Every document GET is appended
to ``Analytics.gets`` as a raw access-log line (path with query string, true to ``Analytics.gets`` as a raw access-log line (path with query string, true
HTTP status, external referer origin, preload flag) and every pagerite.js HTTP status, external referer origin, preload flag, rendered content
activity message from the /_ws WebSocket is appended to ``Analytics.msgs`` language) and every pagerite.js activity message from the /_ws WebSocket is
appended to ``Analytics.msgs``
(navigations ``fr`` -> ``to`` and active reading-time updates). Nothing is (navigations ``fr`` -> ``to`` and active reading-time updates). Nothing is
classified when it is recorded: whether a client turns out to be a reader, classified when it is recorded: whether a client turns out to be a reader,
a crawler or a scanner is decided by ``Store.display()`` from the raw a crawler or a scanner is decided by ``Store.display()`` from the raw
@@ -92,6 +93,9 @@ class Ping(msgspec.Struct, omit_defaults=True):
read: int = 0 read: int = 0
#: Admin client: record but hide everything from the statistics. #: Admin client: record but hide everything from the statistics.
hide: bool = False hide: bool = False
#: Rendered language of the page the activity happened on (the page's
#: ``<html lang>``, sent by pagerite.js).
lang: str = ""
class Get(msgspec.Struct, omit_defaults=True): class Get(msgspec.Struct, omit_defaults=True):
@@ -115,6 +119,9 @@ class Get(msgspec.Struct, omit_defaults=True):
#: never counted as a view/crawler/abuse hit; recorded only so a later #: never counted as a view/crawler/abuse hit; recorded only so a later
#: cache-served navigation can be attributed this GET's status. #: cache-served navigation can be attributed this GET's status.
pre: bool = False pre: bool = False
#: Rendered content language of the served document; "" for
#: non-localized responses (404 probes, reserved paths).
lang: str = ""
class Msg(msgspec.Struct, omit_defaults=True): class Msg(msgspec.Struct, omit_defaults=True):
@@ -135,6 +142,9 @@ class Msg(msgspec.Struct, omit_defaults=True):
to: str = "" to: str = ""
#: Active reading time (seconds) spent on ``fr`` since the last report. #: Active reading time (seconds) spent on ``fr`` since the last report.
read: int = 0 read: int = 0
#: Rendered language reported by the client for the page the activity
#: happened on.
lang: str = ""
class Client(msgspec.Struct, omit_defaults=True): class Client(msgspec.Struct, omit_defaults=True):
@@ -191,6 +201,8 @@ class TrailItem(msgspec.Struct, omit_defaults=True):
``read`` accumulates active reading time (seconds) across the whole ``read`` accumulates active reading time (seconds) across the whole
visit; ``status`` is the most recent HTTP status seen for the target. visit; ``status`` is the most recent HTTP status seen for the target.
A page seen in two rendered languages within one visit (a mid-article
language switch) gets one item per language.
""" """
to: str to: str
@@ -198,6 +210,10 @@ class TrailItem(msgspec.Struct, omit_defaults=True):
read: int = 0 read: int = 0
#: Most recent HTTP status of the response (200 or 404). #: Most recent HTTP status of the response (200 or 404).
status: int = 200 status: int = 200
#: Rendered language of the target: the client's report, for the entry
#: page falling back to its GET's rendered language; "" when unknown
#: (old clients or data from before language recording).
lang: str = ""
class Visit(msgspec.Struct, omit_defaults=True): class Visit(msgspec.Struct, omit_defaults=True):
@@ -206,8 +222,9 @@ class Visit(msgspec.Struct, omit_defaults=True):
``trail`` holds the entry page and everything seen afterwards, keyed by ``trail`` holds the entry page and everything seen afterwards, keyed by
the timestamp of first sight (insertion order = first-seen order); the timestamp of first sight (insertion order = first-seen order);
re-visiting an already seen target updates its item instead of re-visiting an already seen target updates its item instead of
appending. Client metadata is held in ``Analytics.clients`` keyed by appending — unless the client reports a different rendered language for
``client``. it, which appends a distinct item (a mid-article language switch).
Client metadata is held in ``Analytics.clients`` keyed by ``client``.
""" """
start: datetime start: datetime
@@ -241,6 +258,8 @@ class CrawlerHit(msgspec.Struct, omit_defaults=True):
query: str = "" query: str = ""
#: HTTP status of the served response (200 or 404 for content pages). #: HTTP status of the served response (200 or 404 for content pages).
status: int = 200 status: int = 200
#: Rendered content language of the served document (from the GET).
lang: str = ""
class AbuseHit(msgspec.Struct, omit_defaults=True): class AbuseHit(msgspec.Struct, omit_defaults=True):
@@ -319,6 +338,12 @@ class Display(msgspec.Struct, omit_defaults=True):
views: dict[str, dict[str, int]] = {} views: dict[str, dict[str, int]] = {}
#: New visits per 5-minute bucket: bucket ISO -> count (sparse). #: New visits per 5-minute bucket: bucket ISO -> count (sparse).
site_visits: dict[str, int] = {} site_visits: dict[str, int] = {}
#: Site context: true when translation languages are configured, so the
#: viewer can suppress language UI on single-language sites.
multilingual: bool = False
#: The site's primary language (the front page's), so the viewer can
#: skip the primary-language default case.
primary_lang: str = ""
def _bucket(now: datetime) -> str: def _bucket(now: datetime) -> str:
@@ -594,22 +619,25 @@ class Store:
referer: str = "", referer: str = "",
accept_language: str = "", accept_language: str = "",
pre: bool = False, pre: bool = False,
lang: str = "",
) -> bytes | None: ) -> bytes | None:
"""Append one document GET to the raw log. """Append one document GET to the raw log.
``path`` is the full request path, query string included; ``status`` ``path`` is the full request path, query string included; ``status``
the true HTTP status of the response; ``referer`` the raw Referer the true HTTP status of the response; ``referer`` the raw Referer
header (reduced here to an external https origin, "" when internal header (reduced here to an external https origin, "" when internal
or absent); ``pre`` marks idle-time preloads from pagerite.js. or absent); ``pre`` marks idle-time preloads from pagerite.js;
``lang`` the rendered content language of the served document (""
for non-localized responses such as 404 probes and reserved paths).
Returns the client hash when the client record was just created (so Returns the client hash when the client record was just created (so
the caller can schedule async enrichment), else None. the caller can schedule async enrichment), else None.
""" """
lang, country = _parse_accept_language(accept_language) client_lang, country = _parse_accept_language(accept_language)
client_hash = _client_hash(ip, ua, lang) client_hash = _client_hash(ip, ua, client_lang)
new = client_hash not in self.data.clients new = client_hash not in self.data.clients
if new: if new:
self._ensure_client(ip, ua, lang, country=country) self._ensure_client(ip, ua, client_lang, country=country)
self.data.gets.append( self.data.gets.append(
Get( Get(
t=datetime.now(UTC), t=datetime.now(UTC),
@@ -618,6 +646,7 @@ class Store:
status=status, status=status,
ref=_origin(referer) or "", ref=_origin(referer) or "",
pre=pre, pre=pre,
lang=lang,
) )
) )
self._save() self._save()
@@ -632,6 +661,7 @@ class Store:
accept_language: str = "", accept_language: str = "",
hide: bool = False, hide: bool = False,
read: int = 0, read: int = 0,
lang: str = "",
) -> bytes | None: ) -> bytes | None:
"""Append one client activity message (``Ping`` from pagerite.js) to """Append one client activity message (``Ping`` from pagerite.js) to
the raw log. the raw log.
@@ -642,16 +672,18 @@ class Store:
stored raw and filtered at display time, so future rule changes lose stored raw and filtered at display time, so future rule changes lose
nothing. ``hide`` flags the client record as an admin; the message nothing. ``hide`` flags the client record as an admin; the message
itself is recorded normally and hidden at display time like itself is recorded normally and hidden at display time like
everything else the client ever did. everything else the client ever did. ``lang`` is the rendered
language reported by the client for the page the activity happened
on.
Returns the client hash when the client record was just created (so Returns the client hash when the client record was just created (so
the caller can schedule async enrichment), else None. the caller can schedule async enrichment), else None.
""" """
lang, country = _parse_accept_language(accept_language) client_lang, country = _parse_accept_language(accept_language)
client_hash = _client_hash(ip, ua, lang) client_hash = _client_hash(ip, ua, client_lang)
new = client_hash not in self.data.clients new = client_hash not in self.data.clients
if new: if new:
self._ensure_client(ip, ua, lang, country=country) self._ensure_client(ip, ua, client_lang, country=country)
if hide: if hide:
self.data.clients[client_hash].hide = True self.data.clients[client_hash].hide = True
fr = (_internal_path(fr) or "") if fr else "" fr = (_internal_path(fr) or "") if fr else ""
@@ -663,7 +695,14 @@ class Store:
target = _external_target(to) or "" target = _external_target(to) or ""
if target or read > 0: if target or read > 0:
self.data.msgs.append( self.data.msgs.append(
Msg(t=datetime.now(UTC), client=client_hash, fr=fr, to=target, read=read) Msg(
t=datetime.now(UTC),
client=client_hash,
fr=fr,
to=target,
read=read,
lang=lang,
)
) )
if target or read > 0 or hide: if target or read > 0 or hide:
self._save() self._save()
@@ -700,7 +739,13 @@ class Store:
self.data.favicons[origin] = Favicon(file=file, fetched=datetime.now(UTC)) self.data.favicons[origin] = Favicon(file=file, fetched=datetime.now(UTC))
self._save() self._save()
def display(self, in_menu: Callable[[str], bool] | None = None) -> Display: def display(
self,
in_menu: Callable[[str], bool] | None = None,
*,
multilingual: bool = False,
primary_lang: str = "",
) -> Display:
"""Build the viewer payload from the raw events. """Build the viewer payload from the raw events.
All classification happens here, so the stored data is independent All classification happens here, so the stored data is independent
@@ -722,10 +767,17 @@ class Store:
- visits: the remaining messages, grouped per client with a new - visits: the remaining messages, grouped per client with a new
visit after ``_SESSION_GAP`` of inactivity. Trail statuses come visit after ``_SESSION_GAP`` of inactivity. Trail statuses come
from the client's GETs (preloads included — a cache-served from the client's GETs (preloads included — a cache-served
navigation's only GET is its preload); the entry referer and UTM navigation's only GET is its preload); trail languages come from
tags from the GET that loaded the entry page. the client's messages (the entry item falling back to its GET's
rendered language), and a page re-visited in a different rendered
language becomes a distinct trail step. The entry referer and
UTM tags come from the GET that loaded the entry page.
Hidden (admin) clients are excluded from every list and aggregate. Hidden (admin) clients are excluded from every list and aggregate.
``multilingual`` and ``primary_lang`` are site context (translation
languages configured, the front page's primary language) copied
onto the payload so the viewer can suppress language UI on
single-language sites and skip the primary-language default case.
""" """
in_menu = in_menu or (lambda path: False) in_menu = in_menu or (lambda path: False)
data = self.data data = self.data
@@ -836,7 +888,11 @@ class Store:
g.path.split("?", 1)[1] if "?" in g.path else "" g.path.split("?", 1)[1] if "?" in g.path else ""
) )
visit.trail[m.t] = TrailItem( visit.trail[m.t] = TrailItem(
to=m.to, status=status_at(h, m.to, m.t) to=m.to,
status=status_at(h, m.to, m.t),
# The client's report wins; the entry GET fills
# in for old clients that don't send lang.
lang=m.lang or (g.lang if g is not None else ""),
) )
visits.append(visit) visits.append(visit)
else: else:
@@ -844,18 +900,40 @@ class Store:
visit.navs[m.t] = Nav(fr=fr, to=m.to) visit.navs[m.t] = Nav(fr=fr, to=m.to)
status = status_at(h, m.to, m.t) status = status_at(h, m.to, m.t)
# First-seen only: repeat pages and repeated exits # First-seen only: repeat pages and repeated exits
# update the existing trail item instead of appending. # update the existing trail item instead of
# appending — but a repeat in a different rendered
# language (a mid-article language switch) becomes
# a distinct step.
for item in visit.trail.values(): for item in visit.trail.values():
if item.to == m.to: if item.to == m.to:
item.status = status if m.lang and item.lang and m.lang != item.lang:
visit.trail[m.t] = TrailItem(
to=m.to, status=status, lang=m.lang
)
else:
item.status = status
if not item.lang:
item.lang = m.lang
break break
else: else:
visit.trail[m.t] = TrailItem(to=m.to, status=status) visit.trail[m.t] = TrailItem(
to=m.to, status=status, lang=m.lang
)
if m.read > 0 and m.fr and visit is not None: if m.read > 0 and m.fr and visit is not None:
# A page appears in the trail once per language seen:
# land the seconds on the matching-language step when
# the client reports one, else on the first-seen item.
read_item: TrailItem | None = None
for item in visit.trail.values(): for item in visit.trail.values():
if item.to == m.fr: if item.to != m.fr:
item.read += m.read continue
if read_item is None:
read_item = item
if m.lang and item.lang == m.lang:
read_item = item
break break
if read_item is not None:
read_item.read += m.read
last_t = m.t last_t = m.t
# --- crawler hits: document GETs no message matched # --- crawler hits: document GETs no message matched
@@ -889,6 +967,7 @@ class Store:
referer=g.ref, referer=g.ref,
query=query, query=query,
status=g.status, status=g.status,
lang=g.lang,
) )
) )
@@ -912,6 +991,7 @@ class Store:
referer=visit.referer if first else "", referer=visit.referer if first else "",
query=query if first else "", query=query if first else "",
status=item.status, status=item.status,
lang=item.lang,
) )
) )
first = False first = False
@@ -938,6 +1018,8 @@ class Store:
for origin, f in data.favicons.items() for origin, f in data.favicons.items()
if f.file if f.file
}, },
multilingual=multilingual,
primary_lang=primary_lang,
) )
for visit in kept: for visit in kept:
bucket = _bucket(visit.start) bucket = _bucket(visit.start)
@@ -959,6 +1041,14 @@ class Store:
nbuckets[nb] = nbuckets.get(nb, 0) + 1 nbuckets[nb] = nbuckets.get(nb, 0) + 1
return display return display
def display_json(self, in_menu: Callable[[str], bool] | None = None) -> str: def display_json(
self,
in_menu: Callable[[str], bool] | None = None,
*,
multilingual: bool = False,
primary_lang: str = "",
) -> str:
"""The ``display()`` payload as a JSON string for the WebSocket.""" """The ``display()`` payload as a JSON string for the WebSocket."""
return msgspec.json.encode(self.display(in_menu)).decode() return msgspec.json.encode(
self.display(in_menu, multilingual=multilingual, primary_lang=primary_lang)
).decode()
pagerite/pages.py
+4 -3
View File
@@ -152,7 +152,8 @@ async def show_page(request: Request, path: str) -> Response:
if node is not None and node.published and node.chunks is not None: if node is not None and node.published and node.chunks is not None:
# Language selection (docs/localization.md): ?lang= wins when a # Language selection (docs/localization.md): ?lang= wins when a
# translation exists, else header logic. Analytics keep the raw # translation exists, else header logic. Analytics keep the raw
# Accept-Language header regardless of the selection. # Accept-Language header regardless of the selection, and record
# the resolved language as the GET's rendered language.
query_lang = request.query_params.get("lang") query_lang = request.query_params.get("lang")
lang = i18n.select_language( lang = i18n.select_language(
query_lang, query_lang,
@@ -175,7 +176,7 @@ async def show_page(request: Request, path: str) -> Response:
if request.headers.get("if-none-match") == etag: if request.headers.get("if-none-match") == etag:
return Response(status_code=304) return Response(status_code=304)
if _is_trackable_path(path): if _is_trackable_path(path):
_record_get(request) _record_get(request, lang=lang)
return _html_response( return _html_response(
request, request,
"page", "page",
@@ -205,7 +206,7 @@ async def show_page(request: Request, path: str) -> Response:
) )
link_lang = i18n.base_tag(query_lang or "") link_lang = i18n.base_tag(query_lang or "")
if _is_trackable_path(path): if _is_trackable_path(path):
_record_get(request, status=404) _record_get(request, status=404, lang=lang)
return _html_response( return _html_response(
request, request,
"category", "category",
pagerite/tracking.py
+25 -6
View File
@@ -28,7 +28,7 @@ from fastapi import APIRouter, Request, WebSocket, WebSocketDisconnect
from fastapi.responses import Response from fastapi.responses import Response
from uarite import uaparse from uarite import uaparse
from pagerite import analytics from pagerite import analytics, i18n
from pagerite.data import resolve from pagerite.data import resolve
from pagerite.files import _hash_name, file_store from pagerite.files import _hash_name, file_store
from pagerite.state import SITE_URL, _html_response, analytics_store, data from pagerite.state import SITE_URL, _html_response, analytics_store, data
@@ -334,7 +334,7 @@ async def _broadcast_analytics() -> None:
"""Send the current analytics snapshot to every connected WS client.""" """Send the current analytics snapshot to every connected WS client."""
if not _analytics_ws_clients: if not _analytics_ws_clients:
return return
payload = analytics_store.display_json(_in_menu) payload = _display_json()
closed = set() closed = set()
for ws in _analytics_ws_clients: for ws in _analytics_ws_clients:
try: try:
@@ -370,12 +370,29 @@ def _in_menu(path: str) -> bool:
return resolve(data.menu, path.strip("/")) is not None return resolve(data.menu, path.strip("/")) is not None
def _record_get(request: Request, *, status: int = 200) -> None: def _display_json() -> str:
"""The current analytics snapshot as JSON for the admin stream.
Adds the site's language context: ``multilingual`` (translation
languages configured) lets the viewer suppress language UI on
single-language sites, ``primary_lang`` (the front page's) lets it skip
the primary-language default case.
"""
return analytics_store.display_json(
_in_menu,
multilingual=bool(data.translate_langs),
primary_lang=i18n.primary_lang(data.menu, ""),
)
def _record_get(request: Request, *, status: int = 200, lang: str = "") -> None:
"""Record the document GET as one raw access-log line in analytics. """Record the document GET as one raw access-log line in analytics.
Nothing is classified here — the true HTTP status, the full request path Nothing is classified here — the true HTTP status, the full request path
(query included), an external referer origin and the preload flag are (query included), an external referer origin, the preload flag and the
stored, and visitor/crawler/abuse classification happens at display time rendered content language (``lang``, "" for non-localized responses such
as 404 probes and reserved paths) are stored, and
visitor/crawler/abuse classification happens at display time
(see analytics.Store.display). Idle-time preloads from pagerite.js (see analytics.Store.display). Idle-time preloads from pagerite.js
(``x-pagerite-preload`` header) are recorded with ``pre=True``: never (``x-pagerite-preload`` header) are recorded with ``pre=True``: never
counted, but a navigation later served from the in-memory page cache is counted, but a navigation later served from the in-memory page cache is
@@ -403,6 +420,7 @@ def _record_get(request: Request, *, status: int = 200) -> None:
referer=referer, referer=referer,
accept_language=request.headers.get("accept-language", ""), accept_language=request.headers.get("accept-language", ""),
pre=bool(request.headers.get("x-pagerite-preload")), pre=bool(request.headers.get("x-pagerite-preload")),
lang=lang,
) )
if client_hash is not None: if client_hash is not None:
_schedule_client_enrichment([client_hash]) _schedule_client_enrichment([client_hash])
@@ -462,6 +480,7 @@ async def activity_ws(ws: WebSocket) -> None:
accept_language, accept_language,
hide=msg.hide, hide=msg.hide,
read=msg.read, read=msg.read,
lang=msg.lang,
) )
if new_client is not None: if new_client is not None:
_schedule_client_enrichment([new_client]) _schedule_client_enrichment([new_client])
@@ -478,7 +497,7 @@ async def analytics_websocket(ws: WebSocket) -> None:
endpoint. Powers the analytics viewer rendered at /_a. endpoint. Powers the analytics viewer rendered at /_a.
""" """
await ws.accept() await ws.accept()
await ws.send_text(analytics_store.display_json(_in_menu)) await ws.send_text(_display_json())
_analytics_ws_clients.add(ws) _analytics_ws_clients.add(ws)
try: try:
while True: while True: