analytics: record and show the rendered content language of page views
GETs store the resolved page language, activity pings carry <html lang> (including a new ping on in-place language switches, which also keeps the switch's GET out of crawler classification), and trail items/crawler hits surface it. The viewer shows flags discreetly: nothing on single-language sites or primary-language-only rows, one leading flag for uniform visits, transition flags on mixed trails, a flag row per crawler.
This commit is contained in:
+26
-7
@@ -38,6 +38,9 @@ Each `Get` record (one per served document):
|
||||
(`x-pagerite-preload` header): never counted as a view, crawler hit or
|
||||
abuse — recorded only so a navigation later served from the in-memory page
|
||||
cache (which issues no GET at all) can be attributed this GET's status,
|
||||
- `lang` — rendered content language of the served document (the resolved
|
||||
language of a localized page), `""` for non-localized responses (404
|
||||
probes, reserved paths),
|
||||
- `client` — 6-byte blake3 hash referencing `Analytics.clients`.
|
||||
|
||||
304 revalidation responses return before recording and are not logged.
|
||||
@@ -51,7 +54,9 @@ Each `Msg` record (one per pagerite.js activity message over `/_ws`):
|
||||
- `to` — navigation target (validated at record time: internal slug path or
|
||||
external https URL; anything else is dropped — sanitation, not
|
||||
classification),
|
||||
- `read` — active seconds spent on `fr` since the previous report.
|
||||
- `read` — active seconds spent on `fr` since the previous report,
|
||||
- `lang` — rendered language reported by the client for the page the
|
||||
activity happened on (the page's `<html lang>`; `""` from old clients).
|
||||
|
||||
Each `Client` record (shared by every event, keyed by hash):
|
||||
|
||||
@@ -104,7 +109,9 @@ The client (`pagerite.js`) keeps a WebSocket connection to `/_ws` for the
|
||||
whole browsing session and sends activity messages over it — JSON text
|
||||
frames matching the server's `Ping` msgspec struct with the fields `fr`
|
||||
(source path), `to` (navigation target), `read` (active seconds on `fr`
|
||||
since the last report) and `hide`; falsy fields are omitted. One channel
|
||||
since the last report), `lang` (the rendered language of the page the
|
||||
activity happened on — its `<html lang>`) and `hide`; falsy fields are
|
||||
omitted. One channel
|
||||
follows the session, so the activity of a visit stays tied together, and
|
||||
while the user is active the accumulated reading time is flushed every few
|
||||
seconds: the times are incremental, so a disconnection simply leaves the
|
||||
@@ -173,10 +180,16 @@ for misses.
|
||||
(`_SESSION_GAP`). A fresh page load with an already-open visit (second
|
||||
tab) extends it, logging a `(direct)` transition. The visit's trail holds
|
||||
first-seen targets in order; `read` updates accumulate active seconds on
|
||||
the trail item matching `fr`. Each trail item's HTTP status comes from
|
||||
the trail item matching `fr` (preferring the item whose language matches
|
||||
the report, so seconds after a language switch land on the new-language
|
||||
step). Each trail item's HTTP status comes from
|
||||
the client's latest GET for that path — preloads included, which is what
|
||||
allows 404 pages to render red in the viewer even when the navigation
|
||||
itself was served from the page cache. The entry page's referer and
|
||||
itself was served from the page cache. Each trail item also carries the
|
||||
rendered language: the client's report, for the entry page falling back
|
||||
to its GET's rendered language (old clients don't send one); a page
|
||||
re-visited in a different language becomes a distinct trail step instead
|
||||
of merging into the existing item. The entry page's referer and
|
||||
`utm_*` tags come from the GET that loaded it (within 10 s before the
|
||||
first message).
|
||||
- **Crawler hits**: a document GET no activity message matched within
|
||||
@@ -232,6 +245,10 @@ to tell misses from real pages at a glance.
|
||||
The `Display` payload contains the derived `visits`, `crawlers` and `abuse`
|
||||
rows (structs `Visit`/`Nav`/`TrailItem`, `CrawlerHit`, `AbuseHit` — display
|
||||
DTOs only, never persisted), the visible `clients`, the fetched `favicons`,
|
||||
the site language context (`multilingual` — translation languages are
|
||||
configured, so the viewer can suppress language UI on single-language
|
||||
sites — and `primary_lang` — the front page's primary language, so the
|
||||
viewer can skip the primary-language default case),
|
||||
and the aggregates below.
|
||||
|
||||
Each derived `Visit`:
|
||||
@@ -243,8 +260,9 @@ Each derived `Visit`:
|
||||
- `trail` — the entry page and everything seen afterwards, keyed by the
|
||||
timestamp of first sight (insertion order = first-seen order). Each item
|
||||
holds `to` (page path or external exit URL), the accumulated active
|
||||
reading time in seconds (`read`) and the most recent HTTP status seen
|
||||
for the target (`status`),
|
||||
reading time in seconds (`read`), the most recent HTTP status seen
|
||||
for the target (`status`) and the rendered language (`lang`; a page
|
||||
seen in two languages within one visit gets one item per language),
|
||||
- `navs` — every navigation (`fr`, `to`), keyed by its timestamp, repeats
|
||||
included. The aggregates are computed from this log,
|
||||
- `utm` — `utm_*` query parameters from the landing URL, as a dict.
|
||||
@@ -257,7 +275,8 @@ Each derived `CrawlerHit`:
|
||||
- `referer` — external https origin of the request, `""` for direct/none,
|
||||
- `query` — raw query string of the request,
|
||||
- `status` — HTTP status of the served response (200 for a real page, 404
|
||||
for a category placeholder or missing page).
|
||||
for a category placeholder or missing page),
|
||||
- `lang` — rendered content language of the served document (from the GET).
|
||||
|
||||
Each derived `AbuseHit`:
|
||||
|
||||
|
||||
@@ -160,9 +160,15 @@ watch(range, (r) => {
|
||||
|
||||
const clients = computed(() => data.value?.clients || {})
|
||||
const favicons = computed(() => data.value?.favicons || {})
|
||||
const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value))
|
||||
// Site language context from the payload: drives the discreet rendered-
|
||||
// language markers in the visit/crawler rows (multilingual sites only).
|
||||
const site = computed(() => ({
|
||||
multilingual: !!data.value?.multilingual,
|
||||
primaryLang: data.value?.primary_lang || '',
|
||||
}))
|
||||
const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value, site.value))
|
||||
const crawlers = computed(() => rangeData.value?.crawlers || [])
|
||||
const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value))
|
||||
const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value, site.value))
|
||||
const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], clients.value, pageTree.value, now.value))
|
||||
|
||||
</script>
|
||||
@@ -211,7 +217,8 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
|
||||
<td class="trail">
|
||||
<TrailLink v-if="v.refererStep" :step="v.refererStep" :favicons="favicons" @close="$emit('close')" />
|
||||
<span v-if="v.utm && v.utm !== '—'" class="utm-tag small muted" :title="v.utmTitle">{{ v.utm }}</span>
|
||||
<TrailLink v-for="(s, si) in v.trail" :key="si" :step="s" :favicons="favicons" @close="$emit('close')" />
|
||||
<span v-if="v.rowFlag" class="flag" v-html="v.rowFlag" :title="v.rowFlagTitle"></span>
|
||||
<TrailLink v-for="(s, si) in v.trail" :key="si" :step="s" :favicons="favicons" :flags="s.langFlags" @close="$emit('close')" />
|
||||
</td>
|
||||
<VisitorCell
|
||||
:ip="v.ip"
|
||||
@@ -248,6 +255,7 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
|
||||
<td class="trail">
|
||||
<TrailLink v-if="c.refererStep" :step="c.refererStep" :favicons="favicons" @close="$emit('close')" />
|
||||
<TrailLink v-for="(s, si) in c.pages" :key="si" :step="s" :count="s.count" @close="$emit('close')" />
|
||||
<span v-for="(f, fi) in c.readFlags" :key="fi" class="flag" v-html="f.flag" :title="f.name"></span>
|
||||
</td>
|
||||
<VisitorCell
|
||||
:ip="c.ip"
|
||||
@@ -484,6 +492,25 @@ const abuseRows = computed(() => formatAbuseRows(rangeData.value?.abuse || [], c
|
||||
vertical-align: bottom;
|
||||
}
|
||||
|
||||
/* Same flag chip as the visitor cells (VisitorCell.vue); the flags here
|
||||
mark the language the page was read in. */
|
||||
.visit-table .flag {
|
||||
display: inline-flex;
|
||||
width: 18px;
|
||||
height: 12px;
|
||||
border-radius: 2px;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--line);
|
||||
box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.visit-table .flag :deep(svg) {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
display: block;
|
||||
}
|
||||
|
||||
.visit-table .clickable-list {
|
||||
cursor: pointer;
|
||||
max-width: 22rem;
|
||||
|
||||
@@ -6,6 +6,7 @@ const props = defineProps({
|
||||
step: { type: Object, required: true },
|
||||
count: { type: Number, default: 0 },
|
||||
favicons: { type: Object, default: null },
|
||||
flags: { type: Array, default: () => [] },
|
||||
})
|
||||
|
||||
defineEmits(['close'])
|
||||
@@ -38,6 +39,7 @@ const title = computed(() => {
|
||||
<small v-if="count > 1" class="muted">{{ formatCount(count) }}×</small>
|
||||
<img v-if="favicon" class="favicon" :src="favicon" alt="" />
|
||||
<span>{{ step.slug }}</span>
|
||||
<span v-for="(f, fi) in flags" :key="fi" class="flag" v-html="f"></span>
|
||||
</a>
|
||||
</template>
|
||||
|
||||
@@ -48,4 +50,23 @@ const title = computed(() => {
|
||||
margin-right: 0.25em;
|
||||
vertical-align: -0.1em;
|
||||
}
|
||||
|
||||
/* Same flag chip as the visitor cells (VisitorCell.vue). */
|
||||
.flag {
|
||||
display: inline-flex;
|
||||
width: 18px;
|
||||
height: 12px;
|
||||
margin-left: 0.25em;
|
||||
border-radius: 2px;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--line);
|
||||
box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.flag :deep(svg) {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
display: block;
|
||||
}
|
||||
</style>
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
* Formatters and aggregators for summary sections: totals and the recent
|
||||
* visit trail.
|
||||
*/
|
||||
import { flagFor, langName } from '../langs.js'
|
||||
|
||||
/**
|
||||
* IPv4 unchanged, IPv6 returns the /64 network prefix in compact form.
|
||||
@@ -373,7 +374,7 @@ export function mainDomain(host, limit = 24) {
|
||||
* their own site there — rendered with its favicon like visit referers.
|
||||
* ``clients`` maps client hashes to client records.
|
||||
*/
|
||||
export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) {
|
||||
export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now(), site = { multilingual: false, primaryLang: '' }) {
|
||||
const titles = buildTitleMap(pageTree)
|
||||
const groups = new Map()
|
||||
for (const c of crawlers || []) {
|
||||
@@ -384,10 +385,12 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
|
||||
lastStart: 0,
|
||||
referer: '',
|
||||
pages: new Map(),
|
||||
langs: new Set(),
|
||||
}
|
||||
const start = new Date(c.start).getTime()
|
||||
if (start > g.lastStart) g.lastStart = start
|
||||
if (c.referer) g.referer = c.referer
|
||||
if (c.lang) g.langs.add(c.lang)
|
||||
if (c.entry?.startsWith('/')) {
|
||||
const existing = g.pages.get(c.entry) || { count: 0, status: c.status || 200 }
|
||||
existing.count += 1
|
||||
@@ -408,6 +411,11 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
|
||||
const client = g.client || {}
|
||||
const host = client.host || ''
|
||||
const isHost = !!host
|
||||
// Rendered languages read, shown only when they say something the
|
||||
// primary language alone would not (multilingual sites only).
|
||||
const langs = [...g.langs].sort()
|
||||
const showLangs =
|
||||
site.multilingual && (langs.length > 1 || (langs[0] && langs[0] !== site.primaryLang))
|
||||
return {
|
||||
lastSeen: formatWhen(g.lastStart, now),
|
||||
lastSeenIso: formatWhenIso(g.lastStart),
|
||||
@@ -416,6 +424,9 @@ export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now())
|
||||
pages: [...g.pages.entries()]
|
||||
.sort((a, b) => b[1].count - a[1].count)
|
||||
.map(([path, info]) => ({ ...stepOf(path, titles), count: info.count, status: info.status })),
|
||||
readFlags: showLangs
|
||||
? langs.map((l) => ({ flag: flagFor(l), name: langName(l) })).filter((f) => f.flag)
|
||||
: [],
|
||||
ip: client.ip || '',
|
||||
ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip) || client.ip || '—',
|
||||
isHost,
|
||||
@@ -550,23 +561,44 @@ export function formatAbuseRows(abuse, clients, pageTree, now = Date.now()) {
|
||||
* Format raw visit records as rows for a technical table. Returns objects
|
||||
* with display strings; missing values become "—". ``trail`` starts with the
|
||||
* external referer (when present), then the entry page and any further internal
|
||||
* pages or external exit origins. Only the 20 most recent visits are shown.
|
||||
* ``clients`` maps client hashes to client records.
|
||||
* pages or external exit origins; consecutive views of the same page (e.g. a
|
||||
* language switch re-view) merge into one step that keeps the
|
||||
* consecutive-distinct rendered languages, summed read time, and the latest
|
||||
* status. On multilingual sites the rendered languages surface as flag
|
||||
* icons: a visit read entirely in one non-primary language gets ``rowFlag``,
|
||||
* and a visit spanning languages gets per-step ``langFlags`` markers where
|
||||
* the language begins or changes. Only the 20 most recent visits are shown.
|
||||
* ``clients`` maps client hashes to client records; ``site`` carries the
|
||||
* payload's multilingual/primary-language context.
|
||||
*/
|
||||
export function formatVisitRows(visits, clients, pageTree, now = Date.now()) {
|
||||
export function formatVisitRows(visits, clients, pageTree, now = Date.now(), site = { multilingual: false, primaryLang: '' }) {
|
||||
const titles = buildTitleMap(pageTree)
|
||||
return [...(visits || [])].reverse().slice(0, 20).map((v) => {
|
||||
const client = (clients || {})[v.client] || {}
|
||||
const trail = Object.values(v.trail || {})
|
||||
const steps = Object.values(v.trail || {})
|
||||
.map((item) => {
|
||||
const step = stepOf(item.to, titles)
|
||||
if (step) {
|
||||
if (item.read) step.readSeconds = item.read
|
||||
if (item.status) step.status = item.status
|
||||
if (item.lang) step.lang = item.lang
|
||||
}
|
||||
return step
|
||||
})
|
||||
.filter(Boolean)
|
||||
const trail = []
|
||||
for (const step of steps) {
|
||||
const prev = trail[trail.length - 1]
|
||||
if (prev && prev.path === step.path) {
|
||||
if (step.lang && step.lang !== prev.langs[prev.langs.length - 1]) prev.langs.push(step.lang)
|
||||
if (step.readSeconds) prev.readSeconds = (prev.readSeconds || 0) + step.readSeconds
|
||||
if (step.status) prev.status = step.status
|
||||
} else {
|
||||
step.langs = step.lang ? [step.lang] : []
|
||||
trail.push(step)
|
||||
}
|
||||
}
|
||||
const distinctLangs = new Set(trail.flatMap((s) => s.langs))
|
||||
const utmKeys = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content']
|
||||
const utmValues = utmKeys.map((k) => (v.utm || {})[k]).filter(Boolean)
|
||||
const utm = utmValues.length ? utmValues.join(' · ') : ''
|
||||
@@ -576,7 +608,7 @@ export function formatVisitRows(visits, clients, pageTree, now = Date.now()) {
|
||||
const dash = (s) => (s || '—')
|
||||
const host = client.host || ''
|
||||
const isHost = !!host
|
||||
return {
|
||||
const row = {
|
||||
lastSeen: formatWhen(v.start, now),
|
||||
lastSeenIso: formatWhenIso(v.start),
|
||||
lastSeenLocal: formatWhenLocal(v.start),
|
||||
@@ -596,5 +628,27 @@ export function formatVisitRows(visits, clients, pageTree, now = Date.now()) {
|
||||
utm: utm || '—',
|
||||
utmTitle,
|
||||
}
|
||||
if (site.multilingual && distinctLangs.size) {
|
||||
if (distinctLangs.size === 1) {
|
||||
const [tag] = distinctLangs
|
||||
const flag = flagFor(tag)
|
||||
if (flag && tag !== site.primaryLang) {
|
||||
row.rowFlag = flag
|
||||
row.rowFlagTitle = langName(tag)
|
||||
}
|
||||
} else {
|
||||
// Flag the steps where the rendered language begins or changes;
|
||||
// lang-less steps keep the comparison chain going, they never flag.
|
||||
let lastLang = null
|
||||
for (const step of trail) {
|
||||
if (!step.langs.length) continue
|
||||
if (!lastLang || step.langs[step.langs.length - 1] !== lastLang) {
|
||||
step.langFlags = step.langs.map(flagFor).filter(Boolean)
|
||||
}
|
||||
lastLang = step.langs[step.langs.length - 1]
|
||||
}
|
||||
}
|
||||
}
|
||||
return row
|
||||
})
|
||||
}
|
||||
|
||||
@@ -637,6 +637,8 @@ import { reconnectPolicy, socketSlot, watchConnecting } from "./reconnect";
|
||||
if (to) msg.to = to;
|
||||
const secs = Math.round(read / 1000);
|
||||
if (secs > 0) msg.read = secs;
|
||||
const lang = document.documentElement.lang;
|
||||
if (lang) msg.lang = lang;
|
||||
if (!msg.to && !msg.read) return;
|
||||
report(msg);
|
||||
}
|
||||
@@ -815,6 +817,10 @@ import { reconnectPolicy, socketSlot, watchConnecting } from "./reconnect";
|
||||
const y = scrollY; // a language switch is not a navigation: keep scroll
|
||||
await load(currentPath, false);
|
||||
scrollTo(0, y);
|
||||
// Log the switch as a trail event in the new language (load() updated
|
||||
// <html lang>): the ping matches the switch's GET server-side, so it is
|
||||
// not misclassified as a crawler hit.
|
||||
ping({ to: currentPath });
|
||||
});
|
||||
|
||||
// --- Fetch navigation ------------------------------------------------
|
||||
|
||||
+113
-23
@@ -2,8 +2,9 @@
|
||||
|
||||
Raw recording, display-time classification. Every document GET is appended
|
||||
to ``Analytics.gets`` as a raw access-log line (path with query string, true
|
||||
HTTP status, external referer origin, preload flag) and every pagerite.js
|
||||
activity message from the /_ws WebSocket is appended to ``Analytics.msgs``
|
||||
HTTP status, external referer origin, preload flag, rendered content
|
||||
language) and every pagerite.js activity message from the /_ws WebSocket is
|
||||
appended to ``Analytics.msgs``
|
||||
(navigations ``fr`` -> ``to`` and active reading-time updates). Nothing is
|
||||
classified when it is recorded: whether a client turns out to be a reader,
|
||||
a crawler or a scanner is decided by ``Store.display()`` from the raw
|
||||
@@ -92,6 +93,9 @@ class Ping(msgspec.Struct, omit_defaults=True):
|
||||
read: int = 0
|
||||
#: Admin client: record but hide everything from the statistics.
|
||||
hide: bool = False
|
||||
#: Rendered language of the page the activity happened on (the page's
|
||||
#: ``<html lang>``, sent by pagerite.js).
|
||||
lang: str = ""
|
||||
|
||||
|
||||
class Get(msgspec.Struct, omit_defaults=True):
|
||||
@@ -115,6 +119,9 @@ class Get(msgspec.Struct, omit_defaults=True):
|
||||
#: never counted as a view/crawler/abuse hit; recorded only so a later
|
||||
#: cache-served navigation can be attributed this GET's status.
|
||||
pre: bool = False
|
||||
#: Rendered content language of the served document; "" for
|
||||
#: non-localized responses (404 probes, reserved paths).
|
||||
lang: str = ""
|
||||
|
||||
|
||||
class Msg(msgspec.Struct, omit_defaults=True):
|
||||
@@ -135,6 +142,9 @@ class Msg(msgspec.Struct, omit_defaults=True):
|
||||
to: str = ""
|
||||
#: Active reading time (seconds) spent on ``fr`` since the last report.
|
||||
read: int = 0
|
||||
#: Rendered language reported by the client for the page the activity
|
||||
#: happened on.
|
||||
lang: str = ""
|
||||
|
||||
|
||||
class Client(msgspec.Struct, omit_defaults=True):
|
||||
@@ -191,6 +201,8 @@ class TrailItem(msgspec.Struct, omit_defaults=True):
|
||||
|
||||
``read`` accumulates active reading time (seconds) across the whole
|
||||
visit; ``status`` is the most recent HTTP status seen for the target.
|
||||
A page seen in two rendered languages within one visit (a mid-article
|
||||
language switch) gets one item per language.
|
||||
"""
|
||||
|
||||
to: str
|
||||
@@ -198,6 +210,10 @@ class TrailItem(msgspec.Struct, omit_defaults=True):
|
||||
read: int = 0
|
||||
#: Most recent HTTP status of the response (200 or 404).
|
||||
status: int = 200
|
||||
#: Rendered language of the target: the client's report, for the entry
|
||||
#: page falling back to its GET's rendered language; "" when unknown
|
||||
#: (old clients or data from before language recording).
|
||||
lang: str = ""
|
||||
|
||||
|
||||
class Visit(msgspec.Struct, omit_defaults=True):
|
||||
@@ -206,8 +222,9 @@ class Visit(msgspec.Struct, omit_defaults=True):
|
||||
``trail`` holds the entry page and everything seen afterwards, keyed by
|
||||
the timestamp of first sight (insertion order = first-seen order);
|
||||
re-visiting an already seen target updates its item instead of
|
||||
appending. Client metadata is held in ``Analytics.clients`` keyed by
|
||||
``client``.
|
||||
appending — unless the client reports a different rendered language for
|
||||
it, which appends a distinct item (a mid-article language switch).
|
||||
Client metadata is held in ``Analytics.clients`` keyed by ``client``.
|
||||
"""
|
||||
|
||||
start: datetime
|
||||
@@ -241,6 +258,8 @@ class CrawlerHit(msgspec.Struct, omit_defaults=True):
|
||||
query: str = ""
|
||||
#: HTTP status of the served response (200 or 404 for content pages).
|
||||
status: int = 200
|
||||
#: Rendered content language of the served document (from the GET).
|
||||
lang: str = ""
|
||||
|
||||
|
||||
class AbuseHit(msgspec.Struct, omit_defaults=True):
|
||||
@@ -319,6 +338,12 @@ class Display(msgspec.Struct, omit_defaults=True):
|
||||
views: dict[str, dict[str, int]] = {}
|
||||
#: New visits per 5-minute bucket: bucket ISO -> count (sparse).
|
||||
site_visits: dict[str, int] = {}
|
||||
#: Site context: true when translation languages are configured, so the
|
||||
#: viewer can suppress language UI on single-language sites.
|
||||
multilingual: bool = False
|
||||
#: The site's primary language (the front page's), so the viewer can
|
||||
#: skip the primary-language default case.
|
||||
primary_lang: str = ""
|
||||
|
||||
|
||||
def _bucket(now: datetime) -> str:
|
||||
@@ -594,22 +619,25 @@ class Store:
|
||||
referer: str = "",
|
||||
accept_language: str = "",
|
||||
pre: bool = False,
|
||||
lang: str = "",
|
||||
) -> bytes | None:
|
||||
"""Append one document GET to the raw log.
|
||||
|
||||
``path`` is the full request path, query string included; ``status``
|
||||
the true HTTP status of the response; ``referer`` the raw Referer
|
||||
header (reduced here to an external https origin, "" when internal
|
||||
or absent); ``pre`` marks idle-time preloads from pagerite.js.
|
||||
or absent); ``pre`` marks idle-time preloads from pagerite.js;
|
||||
``lang`` the rendered content language of the served document (""
|
||||
for non-localized responses such as 404 probes and reserved paths).
|
||||
|
||||
Returns the client hash when the client record was just created (so
|
||||
the caller can schedule async enrichment), else None.
|
||||
"""
|
||||
lang, country = _parse_accept_language(accept_language)
|
||||
client_hash = _client_hash(ip, ua, lang)
|
||||
client_lang, country = _parse_accept_language(accept_language)
|
||||
client_hash = _client_hash(ip, ua, client_lang)
|
||||
new = client_hash not in self.data.clients
|
||||
if new:
|
||||
self._ensure_client(ip, ua, lang, country=country)
|
||||
self._ensure_client(ip, ua, client_lang, country=country)
|
||||
self.data.gets.append(
|
||||
Get(
|
||||
t=datetime.now(UTC),
|
||||
@@ -618,6 +646,7 @@ class Store:
|
||||
status=status,
|
||||
ref=_origin(referer) or "",
|
||||
pre=pre,
|
||||
lang=lang,
|
||||
)
|
||||
)
|
||||
self._save()
|
||||
@@ -632,6 +661,7 @@ class Store:
|
||||
accept_language: str = "",
|
||||
hide: bool = False,
|
||||
read: int = 0,
|
||||
lang: str = "",
|
||||
) -> bytes | None:
|
||||
"""Append one client activity message (``Ping`` from pagerite.js) to
|
||||
the raw log.
|
||||
@@ -642,16 +672,18 @@ class Store:
|
||||
stored raw and filtered at display time, so future rule changes lose
|
||||
nothing. ``hide`` flags the client record as an admin; the message
|
||||
itself is recorded normally and hidden at display time like
|
||||
everything else the client ever did.
|
||||
everything else the client ever did. ``lang`` is the rendered
|
||||
language reported by the client for the page the activity happened
|
||||
on.
|
||||
|
||||
Returns the client hash when the client record was just created (so
|
||||
the caller can schedule async enrichment), else None.
|
||||
"""
|
||||
lang, country = _parse_accept_language(accept_language)
|
||||
client_hash = _client_hash(ip, ua, lang)
|
||||
client_lang, country = _parse_accept_language(accept_language)
|
||||
client_hash = _client_hash(ip, ua, client_lang)
|
||||
new = client_hash not in self.data.clients
|
||||
if new:
|
||||
self._ensure_client(ip, ua, lang, country=country)
|
||||
self._ensure_client(ip, ua, client_lang, country=country)
|
||||
if hide:
|
||||
self.data.clients[client_hash].hide = True
|
||||
fr = (_internal_path(fr) or "") if fr else ""
|
||||
@@ -663,7 +695,14 @@ class Store:
|
||||
target = _external_target(to) or ""
|
||||
if target or read > 0:
|
||||
self.data.msgs.append(
|
||||
Msg(t=datetime.now(UTC), client=client_hash, fr=fr, to=target, read=read)
|
||||
Msg(
|
||||
t=datetime.now(UTC),
|
||||
client=client_hash,
|
||||
fr=fr,
|
||||
to=target,
|
||||
read=read,
|
||||
lang=lang,
|
||||
)
|
||||
)
|
||||
if target or read > 0 or hide:
|
||||
self._save()
|
||||
@@ -700,7 +739,13 @@ class Store:
|
||||
self.data.favicons[origin] = Favicon(file=file, fetched=datetime.now(UTC))
|
||||
self._save()
|
||||
|
||||
def display(self, in_menu: Callable[[str], bool] | None = None) -> Display:
|
||||
def display(
|
||||
self,
|
||||
in_menu: Callable[[str], bool] | None = None,
|
||||
*,
|
||||
multilingual: bool = False,
|
||||
primary_lang: str = "",
|
||||
) -> Display:
|
||||
"""Build the viewer payload from the raw events.
|
||||
|
||||
All classification happens here, so the stored data is independent
|
||||
@@ -722,10 +767,17 @@ class Store:
|
||||
- visits: the remaining messages, grouped per client with a new
|
||||
visit after ``_SESSION_GAP`` of inactivity. Trail statuses come
|
||||
from the client's GETs (preloads included — a cache-served
|
||||
navigation's only GET is its preload); the entry referer and UTM
|
||||
tags from the GET that loaded the entry page.
|
||||
navigation's only GET is its preload); trail languages come from
|
||||
the client's messages (the entry item falling back to its GET's
|
||||
rendered language), and a page re-visited in a different rendered
|
||||
language becomes a distinct trail step. The entry referer and
|
||||
UTM tags come from the GET that loaded the entry page.
|
||||
|
||||
Hidden (admin) clients are excluded from every list and aggregate.
|
||||
``multilingual`` and ``primary_lang`` are site context (translation
|
||||
languages configured, the front page's primary language) copied
|
||||
onto the payload so the viewer can suppress language UI on
|
||||
single-language sites and skip the primary-language default case.
|
||||
"""
|
||||
in_menu = in_menu or (lambda path: False)
|
||||
data = self.data
|
||||
@@ -836,7 +888,11 @@ class Store:
|
||||
g.path.split("?", 1)[1] if "?" in g.path else ""
|
||||
)
|
||||
visit.trail[m.t] = TrailItem(
|
||||
to=m.to, status=status_at(h, m.to, m.t)
|
||||
to=m.to,
|
||||
status=status_at(h, m.to, m.t),
|
||||
# The client's report wins; the entry GET fills
|
||||
# in for old clients that don't send lang.
|
||||
lang=m.lang or (g.lang if g is not None else ""),
|
||||
)
|
||||
visits.append(visit)
|
||||
else:
|
||||
@@ -844,18 +900,40 @@ class Store:
|
||||
visit.navs[m.t] = Nav(fr=fr, to=m.to)
|
||||
status = status_at(h, m.to, m.t)
|
||||
# First-seen only: repeat pages and repeated exits
|
||||
# update the existing trail item instead of appending.
|
||||
# update the existing trail item instead of
|
||||
# appending — but a repeat in a different rendered
|
||||
# language (a mid-article language switch) becomes
|
||||
# a distinct step.
|
||||
for item in visit.trail.values():
|
||||
if item.to == m.to:
|
||||
if m.lang and item.lang and m.lang != item.lang:
|
||||
visit.trail[m.t] = TrailItem(
|
||||
to=m.to, status=status, lang=m.lang
|
||||
)
|
||||
else:
|
||||
item.status = status
|
||||
if not item.lang:
|
||||
item.lang = m.lang
|
||||
break
|
||||
else:
|
||||
visit.trail[m.t] = TrailItem(to=m.to, status=status)
|
||||
visit.trail[m.t] = TrailItem(
|
||||
to=m.to, status=status, lang=m.lang
|
||||
)
|
||||
if m.read > 0 and m.fr and visit is not None:
|
||||
# A page appears in the trail once per language seen:
|
||||
# land the seconds on the matching-language step when
|
||||
# the client reports one, else on the first-seen item.
|
||||
read_item: TrailItem | None = None
|
||||
for item in visit.trail.values():
|
||||
if item.to == m.fr:
|
||||
item.read += m.read
|
||||
if item.to != m.fr:
|
||||
continue
|
||||
if read_item is None:
|
||||
read_item = item
|
||||
if m.lang and item.lang == m.lang:
|
||||
read_item = item
|
||||
break
|
||||
if read_item is not None:
|
||||
read_item.read += m.read
|
||||
last_t = m.t
|
||||
|
||||
# --- crawler hits: document GETs no message matched
|
||||
@@ -889,6 +967,7 @@ class Store:
|
||||
referer=g.ref,
|
||||
query=query,
|
||||
status=g.status,
|
||||
lang=g.lang,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -912,6 +991,7 @@ class Store:
|
||||
referer=visit.referer if first else "",
|
||||
query=query if first else "",
|
||||
status=item.status,
|
||||
lang=item.lang,
|
||||
)
|
||||
)
|
||||
first = False
|
||||
@@ -938,6 +1018,8 @@ class Store:
|
||||
for origin, f in data.favicons.items()
|
||||
if f.file
|
||||
},
|
||||
multilingual=multilingual,
|
||||
primary_lang=primary_lang,
|
||||
)
|
||||
for visit in kept:
|
||||
bucket = _bucket(visit.start)
|
||||
@@ -959,6 +1041,14 @@ class Store:
|
||||
nbuckets[nb] = nbuckets.get(nb, 0) + 1
|
||||
return display
|
||||
|
||||
def display_json(self, in_menu: Callable[[str], bool] | None = None) -> str:
|
||||
def display_json(
|
||||
self,
|
||||
in_menu: Callable[[str], bool] | None = None,
|
||||
*,
|
||||
multilingual: bool = False,
|
||||
primary_lang: str = "",
|
||||
) -> str:
|
||||
"""The ``display()`` payload as a JSON string for the WebSocket."""
|
||||
return msgspec.json.encode(self.display(in_menu)).decode()
|
||||
return msgspec.json.encode(
|
||||
self.display(in_menu, multilingual=multilingual, primary_lang=primary_lang)
|
||||
).decode()
|
||||
|
||||
+4
-3
@@ -152,7 +152,8 @@ async def show_page(request: Request, path: str) -> Response:
|
||||
if node is not None and node.published and node.chunks is not None:
|
||||
# Language selection (docs/localization.md): ?lang= wins when a
|
||||
# translation exists, else header logic. Analytics keep the raw
|
||||
# Accept-Language header regardless of the selection.
|
||||
# Accept-Language header regardless of the selection, and record
|
||||
# the resolved language as the GET's rendered language.
|
||||
query_lang = request.query_params.get("lang")
|
||||
lang = i18n.select_language(
|
||||
query_lang,
|
||||
@@ -175,7 +176,7 @@ async def show_page(request: Request, path: str) -> Response:
|
||||
if request.headers.get("if-none-match") == etag:
|
||||
return Response(status_code=304)
|
||||
if _is_trackable_path(path):
|
||||
_record_get(request)
|
||||
_record_get(request, lang=lang)
|
||||
return _html_response(
|
||||
request,
|
||||
"page",
|
||||
@@ -205,7 +206,7 @@ async def show_page(request: Request, path: str) -> Response:
|
||||
)
|
||||
link_lang = i18n.base_tag(query_lang or "")
|
||||
if _is_trackable_path(path):
|
||||
_record_get(request, status=404)
|
||||
_record_get(request, status=404, lang=lang)
|
||||
return _html_response(
|
||||
request,
|
||||
"category",
|
||||
|
||||
+25
-6
@@ -28,7 +28,7 @@ from fastapi import APIRouter, Request, WebSocket, WebSocketDisconnect
|
||||
from fastapi.responses import Response
|
||||
from uarite import uaparse
|
||||
|
||||
from pagerite import analytics
|
||||
from pagerite import analytics, i18n
|
||||
from pagerite.data import resolve
|
||||
from pagerite.files import _hash_name, file_store
|
||||
from pagerite.state import SITE_URL, _html_response, analytics_store, data
|
||||
@@ -334,7 +334,7 @@ async def _broadcast_analytics() -> None:
|
||||
"""Send the current analytics snapshot to every connected WS client."""
|
||||
if not _analytics_ws_clients:
|
||||
return
|
||||
payload = analytics_store.display_json(_in_menu)
|
||||
payload = _display_json()
|
||||
closed = set()
|
||||
for ws in _analytics_ws_clients:
|
||||
try:
|
||||
@@ -370,12 +370,29 @@ def _in_menu(path: str) -> bool:
|
||||
return resolve(data.menu, path.strip("/")) is not None
|
||||
|
||||
|
||||
def _record_get(request: Request, *, status: int = 200) -> None:
|
||||
def _display_json() -> str:
|
||||
"""The current analytics snapshot as JSON for the admin stream.
|
||||
|
||||
Adds the site's language context: ``multilingual`` (translation
|
||||
languages configured) lets the viewer suppress language UI on
|
||||
single-language sites, ``primary_lang`` (the front page's) lets it skip
|
||||
the primary-language default case.
|
||||
"""
|
||||
return analytics_store.display_json(
|
||||
_in_menu,
|
||||
multilingual=bool(data.translate_langs),
|
||||
primary_lang=i18n.primary_lang(data.menu, ""),
|
||||
)
|
||||
|
||||
|
||||
def _record_get(request: Request, *, status: int = 200, lang: str = "") -> None:
|
||||
"""Record the document GET as one raw access-log line in analytics.
|
||||
|
||||
Nothing is classified here — the true HTTP status, the full request path
|
||||
(query included), an external referer origin and the preload flag are
|
||||
stored, and visitor/crawler/abuse classification happens at display time
|
||||
(query included), an external referer origin, the preload flag and the
|
||||
rendered content language (``lang``, "" for non-localized responses such
|
||||
as 404 probes and reserved paths) are stored, and
|
||||
visitor/crawler/abuse classification happens at display time
|
||||
(see analytics.Store.display). Idle-time preloads from pagerite.js
|
||||
(``x-pagerite-preload`` header) are recorded with ``pre=True``: never
|
||||
counted, but a navigation later served from the in-memory page cache is
|
||||
@@ -403,6 +420,7 @@ def _record_get(request: Request, *, status: int = 200) -> None:
|
||||
referer=referer,
|
||||
accept_language=request.headers.get("accept-language", ""),
|
||||
pre=bool(request.headers.get("x-pagerite-preload")),
|
||||
lang=lang,
|
||||
)
|
||||
if client_hash is not None:
|
||||
_schedule_client_enrichment([client_hash])
|
||||
@@ -462,6 +480,7 @@ async def activity_ws(ws: WebSocket) -> None:
|
||||
accept_language,
|
||||
hide=msg.hide,
|
||||
read=msg.read,
|
||||
lang=msg.lang,
|
||||
)
|
||||
if new_client is not None:
|
||||
_schedule_client_enrichment([new_client])
|
||||
@@ -478,7 +497,7 @@ async def analytics_websocket(ws: WebSocket) -> None:
|
||||
endpoint. Powers the analytics viewer rendered at /_a.
|
||||
"""
|
||||
await ws.accept()
|
||||
await ws.send_text(analytics_store.display_json(_in_menu))
|
||||
await ws.send_text(_display_json())
|
||||
_analytics_ws_clients.add(ws)
|
||||
try:
|
||||
while True:
|
||||
|
||||
Reference in New Issue
Block a user