LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

analytics: 24h day view with bar chart for precise realtime stats. Tables redesigned with cleaner layout. Tracking article read times. Adjust connection graph visualizations by time range. Other cleanup and supporting systems.

Browse Source
This commit is contained in:
LeoVasanko
2026-08-21 20:14:04 +00:00
parent f341d22aa0
commit 6eaa1c1a8b
11 changed files with 1119 additions and 159 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/analytics.md
+40 -5
View File
@@ -38,10 +38,17 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
links are stored by full URL so several links to the same domain remain links are stored by full URL so several links to the same domain remain
distinct. distinct.
- **Excluded**: back/forward (popstate) navigations, navigation involving - **Excluded**: back/forward (popstate) navigations, navigation involving
the analytics page itself (`/_a`), and everything while the user is known to the analytics page itself (`/_a`), and everything while the user has the
be an admin *and SSO is actually in use* — with no auth proxy (dev/test) editor open (`body.editing`). Admin noise, not visits.
"admin" is everyone's state, so the gate is off and everything is recorded — - **Admins**: when SSO is in use and the session is known to be an admin,
or has the editor open (`body.editing`). Admin noise, not visits. the client still pings but adds `hide=1`. The server then records
nothing — and if the same (IP, UA) session already had a visit from
before logging in, that visit is removed from the JSON along with the
counts recorded when it was created (site visit, entry view, entry
transition). Views/transitions logged by later pings inside such a visit
lack per-event timestamps and are left as-is. With no auth proxy
(dev/test) "admin" is everyone's state, so `hide` stays 0 and everything
is recorded.
- The server validates `to`: internal paths must be valid slug paths - The server validates `to`: internal paths must be valid slug paths
("/" or `[a-z0-9_-]` segments), external ones are re-derived to the ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the
https origin and accepted only when the client sent exactly that. https origin and accepted only when the client sent exactly that.
@@ -69,6 +76,19 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
hits are grouped by the same (IP, User-Agent) pair and shown as a trail of hits are grouped by the same (IP, User-Agent) pair and shown as a trail of
internal pages that crawler visited; the crawler table lists the most active internal pages that crawler visited; the crawler table lists the most active
crawlers first rather than the most recent hits. crawlers first rather than the most recent hits.
- **Abuse (scanner) hits**: a 404 for a telltale path — any URL segment
starting with a dot (`/.env`, `/.git/config`) or ending in `.php` —
classifies the source IP as abuse immediately, and ten plain 404s from one
IP do too. Classification reclassifies history: all earlier crawler hits
from that IP (persisted and pending) move to the `abuse` list, so a
random-UA scanner no longer pollutes the crawler stats of the legitimate
bot it impersonates. Once classified, every document GET and 404 from the
IP is recorded as an abuse hit with the full request path (query string
included), and its pings are ignored. The classified IP set (`abuse_ips`)
is persisted in the JSON file; the plain-404 counters are RAM-only. In the
viewer, abuse hits are grouped by IP (never by UA — scanners randomize
theirs) in a separate "Abuse" table listing the full paths probed and the
raw User-Agent strings, one per line, with click-to-copy full lists.
## Visits and sessions ## Visits and sessions
@@ -111,7 +131,22 @@ Each `CrawlerHit` record:
- `referer` — external https origin of the request, `""` for direct/none, - `referer` — external https origin of the request, `""` for direct/none,
- `query` — raw query string of the request. - `query` — raw query string of the request.
Crawler hits are grouped by User-Agent in the analytics viewer. Each `AbuseHit` record:
- `start` — timestamp of the request,
- `path` — full request path including the query string (e.g. `/.env?x=1`),
- `ip` — IP address (the grouping key for abusers),
- `ua` — raw `User-Agent` header,
- `ua_pretty` — compact display form of the UA when parsable,
- `flag` — true for the path that triggered abuse classification (telltale
path or the 404 that crossed the threshold),
- `is_404` — true for 404 responses, false for document GETs from the
abuser.
Crawler hits are grouped by (IP, User-Agent) in the analytics viewer; abuse
hits are grouped by IP alone. In the Abuse table paths are listed in access
order, oldest first, with flagged paths lifted to the top, followed by other
404s and then document GETs.
## Aggregates ## Aggregates
frontend/src/AnalyticsView.vue
+260 -45
View File
@@ -9,6 +9,8 @@ import { RANGES } from './analytics/time.js'
import { import {
calcTotalViews, calcTotalViews,
copyIp, copyIp,
copyList,
formatAbuseRows,
formatCrawlerRows, formatCrawlerRows,
formatVisitRows, formatVisitRows,
} from './analytics/format.js' } from './analytics/format.js'
@@ -20,6 +22,8 @@ const props = defineProps({
initialRange: { type: String, default: 'week' }, initialRange: { type: String, default: 'week' },
}) })
const ABUSE_MAX_LINES = 5
const data = ref(null) const data = ref(null)
const pageTree = ref(null) const pageTree = ref(null)
const error = ref('') const error = ref('')
@@ -52,7 +56,7 @@ function connectAnalytics() {
onMounted(async () => { onMounted(async () => {
connectAnalytics() connectAnalytics()
now.value = Date.now() now.value = Date.now()
timeInterval = setInterval(() => { now.value = Date.now() }, 30000) timeInterval = setInterval(() => { now.value = Date.now() }, 1000)
// The site tree for the transition map (all pages in menu order). Not // The site tree for the transition map (all pages in menu order). Not
// fatal: without it the map just narrows to pages seen in transitions. // fatal: without it the map just narrows to pages seen in transitions.
try { try {
@@ -86,6 +90,7 @@ watch(range, (r) => {
const visitRows = computed(() => formatVisitRows(visits.value, pageTree.value, now.value)) const visitRows = computed(() => formatVisitRows(visits.value, pageTree.value, now.value))
const crawlers = computed(() => data.value?.crawlers || []) const crawlers = computed(() => data.value?.crawlers || [])
const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, pageTree.value, now.value)) const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, pageTree.value, now.value))
const abuseRows = computed(() => formatAbuseRows(data.value?.abuse || [], now.value))
function flagSvg(code) { function flagSvg(code) {
return flagSvgs[code?.toUpperCase()] || '' return flagSvgs[code?.toUpperCase()] || ''
@@ -131,20 +136,24 @@ function countryName(code) {
<table class="visit-table"> <table class="visit-table">
<thead> <thead>
<tr> <tr>
<th>when</th>
<th>trail</th> <th>trail</th>
<th>referer</th> <th>visitor</th>
<th>ip</th> <th class="last-seen">last seen</th>
<th>lang</th>
<th>country</th>
<th>ua</th>
<th>utm</th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
<tr v-for="(v, i) in visitRows" :key="i"> <tr v-for="(v, i) in visitRows" :key="i">
<td class="when" :title="v.whenTooltip">{{ v.when }}</td>
<td class="trail"> <td class="trail">
<template v-if="v.refererStep">
<a class="trail-link"
:href="v.refererStep.path"
:title="v.refererStep.title"
:target="v.refererStep.external ? '_blank' : undefined"
:rel="v.refererStep.external ? 'noopener' : undefined">
{{ v.refererStep.slug }}
</a>
</template>
<span v-if="v.utm && v.utm !== '—'" class="utm-tag"><small class="muted">{{ v.utm }}</small></span>
<a v-for="(s, si) in v.trail" :key="si" <a v-for="(s, si) in v.trail" :key="si"
:href="s.path" :title="s.title" :href="s.path" :title="s.title"
:target="s.external ? '_blank' : undefined" :target="s.external ? '_blank' : undefined"
@@ -153,21 +162,27 @@ function countryName(code) {
{{ s.slug }} {{ s.slug }}
</a> </a>
</td> </td>
<td>{{ v.referer }}</td> <td class="ip-locale-cell" :class="{ 'host-cell': v.isHost }">
<td> <div class="ip-locale-rows">
<span class="clickable-ip" <div class="ip-locale-row">
:title="`Click to copy full IP: ${v.ip}`" <div class="locale-line">
@click="copyIp(v.ip)">{{ v.ipDisplay }}</span> <span v-if="flagSvg(v.country)" class="flag" v-html="flagSvg(v.country)" :title="countryName(v.country) || v.country"></span>
<template v-if="v.city && v.city !== '—'"><small class="city-name">{{ v.city }}</small></template>
<template v-else-if="!flagSvg(v.country)">—</template>
</div>
<div class="ip-line"><span class="clickable-ip"
:title="v.ip"
@click="copyIp(v.ip, $event)">{{ v.ipDisplay }}</span></div>
</div>
<div class="ip-locale-row">
<div class="ua-line"><small class="muted" :title="v.uaRaw">{{ v.ua }}</small></div>
<div v-if="v.lang && v.lang !== '—'" class="locale-lang"><small class="muted">{{ v.langDisplay }}</small></div>
</div>
</div>
</td> </td>
<td>{{ v.lang }}</td> <td class="last-seen"
<td class="country"> :title="v.lastSeenLocal"
<span v-if="flagSvg(v.country)" class="flag" v-html="flagSvg(v.country)" :title="countryName(v.country) || v.country"></span> @click="copyList(v.lastSeenIso, $event)">{{ v.lastSeen }}</td>
<template v-if="v.city !== '—'">{{ countryName(v.country) || v.country }}<br><small class="muted">{{ v.city }}</small></template>
<template v-else-if="v.country !== '—'">{{ countryName(v.country) || v.country }}</template>
<template v-else>—</template>
</td>
<td class="ua" :title="v.uaRaw">{{ v.ua }}</td>
<td>{{ v.utm }}</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@@ -181,15 +196,13 @@ function countryName(code) {
<table class="visit-table"> <table class="visit-table">
<thead> <thead>
<tr> <tr>
<th>when</th>
<th>pages</th> <th>pages</th>
<th>ip</th> <th>ip / ua</th>
<th>ua</th> <th class="last-seen">last seen</th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
<tr v-for="(c, i) in crawlerRows" :key="i"> <tr v-for="(c, i) in crawlerRows" :key="i">
<td class="when" :title="c.whenTooltip">{{ c.when }}</td>
<td class="trail"> <td class="trail">
<a v-for="(s, si) in c.pages" :key="si" <a v-for="(s, si) in c.pages" :key="si"
:href="s.path" :title="`${s.title}${s.count > 1 ? ` (${s.count} hits)` : ''}`" :href="s.path" :title="`${s.title}${s.count > 1 ? ` (${s.count} hits)` : ''}`"
@@ -197,18 +210,68 @@ function countryName(code) {
<small v-if="s.count > 1" class="muted">{{ s.count }}×</small>{{ s.slug }} <small v-if="s.count > 1" class="muted">{{ s.count }}×</small>{{ s.slug }}
</a> </a>
</td> </td>
<td> <td class="ip-ua-cell">
<span class="clickable-ip" <div><span class="clickable-ip"
:title="`Click to copy full IP: ${c.ip}`" :title="c.ip"
@click="copyIp(c.ip)">{{ c.ipDisplay }}</span> @click="copyIp(c.ip, $event)">{{ c.ipDisplay }}</span></div>
<div class="ua-line"><small class="muted" :title="c.uaRaw">{{ c.ua }}</small></div>
</td> </td>
<td class="ua" :title="c.uaRaw">{{ c.ua }}</td> <td class="last-seen"
:title="c.lastSeenLocal"
@click="copyList(c.lastSeenIso, $event)">{{ c.lastSeen }}</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
</div> </div>
<p v-else class="empty">no crawler hits recorded yet</p> <p v-else class="empty">no crawler hits recorded yet</p>
</section> </section>
<section v-if="abuseRows.length">
<h2>Abuse</h2>
<div class="visit-table-wrap">
<table class="visit-table">
<thead>
<tr>
<th>paths</th>
<th>ip / uas</th>
<th class="last-seen">last seen</th>
</tr>
</thead>
<tbody>
<tr v-for="(a, i) in abuseRows" :key="i">
<td class="trail abuse-list clickable-list"
@click="copyList(a.allPaths, $event)">
<div v-for="(p, pi) in a.paths.slice(0, ABUSE_MAX_LINES)" :key="pi"
class="list-line">
{{ p.path }}
</div>
<div v-if="a.paths.length > ABUSE_MAX_LINES" class="list-line">
<small class="muted">+{{ a.paths.length - ABUSE_MAX_LINES }} more</small>
</div>
</td>
<td class="ip-ua-cell">
<div><span class="clickable-ip"
:title="a.ip"
@click="copyIp(a.ip, $event)">{{ a.ipDisplay }}</span></div>
<div class="abuse-uas-list clickable-list"
@click="copyList(a.allUas, $event)">
<div v-for="(u, ui) in a.uas.slice(0, ABUSE_MAX_LINES)" :key="ui"
class="list-line">
<small v-if="u.count > 1" class="muted">{{ u.count }}×</small>{{ u.ua }}
</div>
<div v-if="a.uas.length > ABUSE_MAX_LINES" class="list-line">
<small class="muted">+{{ a.uas.length - ABUSE_MAX_LINES }} more</small>
</div>
</div>
</td>
<td class="last-seen"
:title="a.lastSeenLocal"
@click="copyList(a.lastSeenIso, $event)">{{ a.lastSeen }}</td>
</tr>
</tbody>
</table>
</div>
</section>
</template> </template>
</div> </div>
</div> </div>
@@ -296,7 +359,6 @@ function countryName(code) {
.visit-table { .visit-table {
width: 100%; width: 100%;
border-collapse: collapse; border-collapse: collapse;
font-family: monospace;
font-size: 0.82rem; font-size: 0.82rem;
line-height: 1.3; line-height: 1.3;
} }
@@ -318,9 +380,16 @@ function countryName(code) {
background: var(--bg, Canvas); background: var(--bg, Canvas);
} }
.visit-table .when { .visit-table .last-seen {
width: 7.5rem;
text-align: right;
white-space: nowrap; white-space: nowrap;
color: var(--muted); color: var(--muted);
cursor: pointer;
}
.visit-table .last-seen:hover {
color: var(--accent);
} }
.visit-table .trail { .visit-table .trail {
@@ -328,17 +397,50 @@ function countryName(code) {
overflow-wrap: break-word; overflow-wrap: break-word;
} }
.visit-table .trail a { .visit-table .trail a,
.visit-table .trail-link {
display: inline-block;
max-width: 8rem;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
color: var(--text); color: var(--text);
text-decoration: none; text-decoration: none;
vertical-align: bottom;
} }
.visit-table .trail a:hover { color: var(--accent); } .visit-table .trail a:hover,
.visit-table .trail-link:hover { color: var(--accent); }
.visit-table .trail a + a { .visit-table .trail > * + * {
margin-left: 0.5rem; margin-left: 0.5rem;
} }
.visit-table .utm-tag {
display: inline-block;
color: var(--muted);
}
.visit-table .clickable-list {
cursor: pointer;
max-width: 22rem;
}
.visit-table .clickable-list .list-line {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
line-height: 1.35;
}
.visit-table .clickable-list .list-line + .list-line {
margin-top: 0.15rem;
}
.visit-table .ua.abuse-uas {
max-width: 24rem;
}
.visit-table .trail small, .visit-table .trail small,
.visit-table small.muted { .visit-table small.muted {
color: var(--muted); color: var(--muted);
@@ -346,23 +448,130 @@ function countryName(code) {
} }
.visit-table .clickable-ip { .visit-table .clickable-ip {
cursor: pointer; font-size: 0.75em;
text-decoration: underline;
text-decoration-style: dotted;
} }
.visit-table .clickable-ip:hover { .visit-table .clickable-ip,
.visit-table .clickable-list,
.visit-table .last-seen {
cursor: pointer;
position: relative;
}
.visit-table .clickable-ip:hover,
.visit-table .clickable-list:hover,
.visit-table .last-seen:hover {
color: var(--accent); color: var(--accent);
} }
.visit-table .ua { .visit-table .ip-locale-cell {
max-width: 18rem; width: 36ch;
max-width: 36ch;
overflow: hidden;
text-overflow: ellipsis;
}
.visit-table .ip-ua-cell {
width: 22ch;
max-width: 22ch;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.visit-table .host-cell {
text-align: right;
}
.visit-table .ip-locale-rows {
display: flex;
flex-direction: column;
gap: 0.15rem;
}
.visit-table .ip-locale-row {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.5rem;
}
.visit-table .ip-locale-row > * {
min-width: 0;
}
.visit-table .ip-locale-row .locale-line,
.visit-table .ip-locale-row .ip-line,
.visit-table .ip-locale-row .ua-line {
flex: 1 1 auto;
overflow: hidden; overflow: hidden;
text-overflow: ellipsis; text-overflow: ellipsis;
white-space: nowrap; white-space: nowrap;
} }
.visit-table .country .flag { .visit-table .ip-locale-row .locale-lang {
flex: 0 1 auto;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
text-align: right;
}
.visit-table .ip-locale-row .locale-line {
text-align: left;
}
.visit-table .ip-locale-row .ip-line {
text-align: right;
}
.visit-table .ip-locale-row .ua-line {
text-align: left;
}
.visit-table .locale-line {
display: flex;
align-items: center;
gap: 0.3rem;
}
.visit-table .city-name {
display: inline-block;
max-width: 10ch;
font-size: 0.75em;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
vertical-align: middle;
}
.visit-table .ua-line {
text-align: right;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.visit-table .abuse-uas-list {
text-align: right;
}
.visit-table .copy-popup {
position: absolute;
bottom: calc(100% + 0.25rem);
left: 50%;
transform: translateX(-50%);
padding: 0.15rem 0.4rem;
background: var(--text, CanvasText);
color: var(--bg, Canvas);
border-radius: 0.25rem;
font-size: 0.75rem;
white-space: nowrap;
pointer-events: none;
z-index: 10;
}
.visit-table .locale-line .flag {
display: inline-flex; display: inline-flex;
width: 18px; width: 18px;
height: 12px; height: 12px;
@@ -370,14 +579,20 @@ function countryName(code) {
overflow: hidden; overflow: hidden;
border: 1px solid var(--line); border: 1px solid var(--line);
box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset; box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
vertical-align: middle;
} }
.visit-table .country .flag :deep(svg) { .visit-table .locale-line .flag :deep(svg) {
width: 100%; width: 100%;
height: 100%; height: 100%;
display: block; display: block;
} }
.visit-table .locale-line .city-name {
margin-left: 0.3rem;
vertical-align: middle;
}
.crawler-top-uas { .crawler-top-uas {
font-size: 0.9rem; font-size: 0.9rem;
margin-bottom: 0.6rem; margin-bottom: 0.6rem;
frontend/src/TransitionGraph.vue
+59 -11
View File
@@ -7,7 +7,7 @@
* range, exactly like the charts and per-page views do. * range, exactly like the charts and per-page views do.
*/ */
import { computed, onBeforeUnmount, shallowRef, watch } from 'vue' import { computed, onBeforeUnmount, shallowRef, watch } from 'vue'
import { rangeWindow } from './analytics/time.js' import { rangeWindow, WEEK } from './analytics/time.js'
import { import {
TNODE_R, TNODE_R,
BEAD_R, BEAD_R,
@@ -25,6 +25,19 @@ const props = defineProps({
const window = computed(() => rangeWindow(props.range)) const window = computed(() => rangeWindow(props.range))
const visualScale = computed(() => {
const { t0, t1 } = window.value
if (t0 != null && t1 != null) return WEEK / (t1 - t0)
// 'all': scale by the actual data span.
const times = new Set()
for (const buckets of Object.values(props.data?.views || {})) {
for (const k of Object.keys(buckets)) times.add(Date.parse(k))
}
const arr = [...times]
if (arr.length < 2) return 1
return WEEK / (Math.max(...arr) - Math.min(...arr))
})
const filteredData = computed(() => { const filteredData = computed(() => {
if (!props.data) return null if (!props.data) return null
const { t0, t1 } = window.value const { t0, t1 } = window.value
@@ -36,7 +49,7 @@ const filteredData = computed(() => {
const graph = computed(() => const graph = computed(() =>
filteredData.value filteredData.value
? buildTransitionGraph(filteredData.value, props.pageTree) ? buildTransitionGraph(filteredData.value, props.pageTree, props.data?.visits, visualScale.value)
: null, : null,
) )
@@ -47,16 +60,24 @@ const graph = computed(() =>
const beads = shallowRef([]) const beads = shallowRef([])
let rafId = 0 let rafId = 0
const MAX_BEAD_RATE = 120 // upper bound on total beads per second
const startBeads = (flows) => { const startBeads = (flows) => {
cancelAnimationFrame(rafId) cancelAnimationFrame(rafId)
beads.value = [] beads.value = []
if (!flows?.length) return if (!flows?.length) return
if (matchMedia('(prefers-reduced-motion: reduce)').matches) return if (matchMedia('(prefers-reduced-motion: reduce)').matches) return
// Cap the total bead emission rate so a busy range cannot spawn enough
// beads to kill the page. Existing per-range time scaling is preserved;
// this is only a proportional emergency throttle when the limit is hit.
const totalRate = flows.reduce((s, f) => s + 1 / f.interval, 0)
const scale = totalRate > MAX_BEAD_RATE ? MAX_BEAD_RATE / totalRate : 1
const live = [] // { flow, t0 } — one entry per bead in flight const live = [] // { flow, t0 } — one entry per bead in flight
const now = performance.now() const now = performance.now()
const emitters = flows.map((flow) => { const emitters = flows.map((flow) => {
const interval = flow.interval * 1000 const interval = (flow.interval / scale) * 1000
// Pre-fill the traversal with evenly spaced beads (random phase), so // Pre-fill the traversal with evenly spaced beads (random phase), so
// the flow appears already running instead of starting empty. // the flow appears already running instead of starting empty.
const phase = Math.random() * interval const phase = Math.random() * interval
@@ -100,10 +121,14 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
<section v-if="graph"> <section v-if="graph">
<svg class="tmap" :viewBox="`${graph.bounds.x0} ${graph.bounds.y0} ${graph.bounds.x1 - graph.bounds.x0} ${graph.bounds.y1 - graph.bounds.y0}`" <svg class="tmap" :viewBox="`${graph.bounds.x0} ${graph.bounds.y0} ${graph.bounds.x1 - graph.bounds.x0} ${graph.bounds.y1 - graph.bounds.y0}`"
role="img" aria-label="map of transitions between pages"> role="img" aria-label="map of transitions between pages">
<defs>
<!-- Unit-radius circle; only the portion near the bottom is used. -->
<path id="tnode-label-arc" d="M 0,-1 A 1,1 0 1,0 0,1 A 1,1 0 1,0 -0.001,-1" />
</defs>
<path v-for="(a, i) in graph.arcs" :key="'a' + i" <path v-for="(a, i) in graph.arcs" :key="'a' + i"
:d="a.d" class="tarc" /> :d="a.d" class="tarc" />
<path v-for="(e, i) in graph.edges" :key="'e' + i" <path v-for="(e, i) in graph.edges" :key="'e' + i"
:d="e.d" class="tconn"> :d="e.d" :class="['tconn', e.external && 'tconn-exit']">
<title>{{ e.title }}</title> <title>{{ e.title }}</title>
</path> </path>
<circle v-for="(b, i) in beads" :key="'b' + i" <circle v-for="(b, i) in beads" :key="'b' + i"
@@ -112,16 +137,32 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
<a :href="x.path" target="_blank" rel="noopener" :title="x.path"> <a :href="x.path" target="_blank" rel="noopener" :title="x.path">
<circle :cx="x.x" :cy="x.y" :r="x.r" <circle :cx="x.x" :cy="x.y" :r="x.r"
:class="['txnode', x.kind === 'source' ? 'txnode-source' : 'txnode-exit']" /> :class="['txnode', x.kind === 'source' ? 'txnode-source' : 'txnode-exit']" />
<text :x="x.x" :y="x.y - 2" class="tnodeslug">{{ x.label }}</text> <text :transform="`translate(${x.x}, ${x.y}) scale(${x.r - 4})`" class="tnodeslug" :style="{ '--node-r': x.r - 4 }">
<text :x="x.x" :y="x.y + 12" class="tnodecount">{{ x.count }}</text> <textPath href="#tnode-label-arc" startOffset="50%" text-anchor="middle" side="right">{{ x.label }}</textPath>
</text>
<text :x="x.x" :y="x.y + 4" class="tnodecount">{{ x.count }}</text>
</a> </a>
</g> </g>
<g v-for="n in graph.nodes" :key="n.path"> <g v-for="n in graph.nodes" :key="n.path">
<a :href="n.path" :title="n.title"> <a v-if="!n.hidden" :href="n.path" :title="n.title">
<circle :cx="n.x" :cy="n.y" :r="TNODE_R" class="tnode" /> <circle :cx="n.x" :cy="n.y" :r="TNODE_R" class="tnode" />
<text :x="n.x" :y="n.y - 2" class="tnodeslug">{{ n.label }}</text> <text :transform="`translate(${n.x}, ${n.y}) scale(${TNODE_R - 4})`" class="tnodeslug" :style="{ '--node-r': TNODE_R - 4 }">
<text :x="n.x" :y="n.y + 12" class="tnodecount">{{ n.views }}</text> <textPath href="#tnode-label-arc" startOffset="50%" text-anchor="middle" side="right">{{ n.label }}</textPath>
</text>
<text :x="n.x" :y="n.y + 4" class="tnodecount">
{{ n.readMin ? `${n.views}×${n.readMin}m` : n.views }}
</text>
</a> </a>
<template v-else>
<text :x="n.x" :y="n.y"
:transform="`rotate(${n.angle * 180 / Math.PI}, ${n.x}, ${n.y})`"
class="tnodehidden" text-anchor="start" dominant-baseline="middle">➤</text>
<text :x="n.x + Math.cos(n.angle) * 10"
:y="n.y + Math.sin(n.angle) * 10"
:transform="`rotate(${(n.angle + (Math.cos(n.angle) < 0 ? Math.PI : 0)) * 180 / Math.PI}, ${n.x + Math.cos(n.angle) * 10}, ${n.y + Math.sin(n.angle) * 10})`"
:text-anchor="Math.cos(n.angle) < 0 ? 'end' : 'start'"
class="tnodehidden" dominant-baseline="middle">{{ n.label }}</text>
</template>
</g> </g>
</svg> </svg>
</section> </section>
@@ -139,6 +180,9 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
fill: var(--accent); fill: var(--accent);
opacity: 0.4; /* uniform, not strength-encoded: width carries that */ opacity: 0.4; /* uniform, not strength-encoded: width carries that */
} }
.tmap .tconn-exit {
fill: var(--text);
}
.tmap .tbead { .tmap .tbead {
fill: var(--accent); fill: var(--accent);
opacity: 0.85; opacity: 0.85;
@@ -149,7 +193,7 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
stroke-width: 1.5; stroke-width: 1.5;
} }
.tmap .txnode-source { stroke: var(--text); } .tmap .txnode-source { stroke: var(--text); }
.tmap .txnode-exit { stroke: var(--muted); } .tmap .txnode-exit { stroke: var(--text); }
.tmap .tarc { .tmap .tarc {
fill: none; fill: none;
stroke: var(--line); stroke: var(--line);
@@ -162,7 +206,7 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
} }
.tmap .tnodeslug { .tmap .tnodeslug {
fill: var(--text); fill: var(--text);
font-size: 11px; font-size: calc(11px / var(--node-r, 34));
text-anchor: middle; text-anchor: middle;
} }
.tmap a { cursor: pointer; } .tmap a { cursor: pointer; }
@@ -172,6 +216,10 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
font-size: 10px; font-size: 10px;
text-anchor: middle; text-anchor: middle;
} }
.tmap .tnodehidden {
fill: var(--text);
font-size: 9px;
}
section { margin-top: 1.8rem; } section { margin-top: 1.8rem; }
</style> </style>
frontend/src/VisitorCharts.vue
+42 -15
View File
@@ -2,10 +2,12 @@
/** /**
* Visitor and page-view smoothed curves for a single shared time range. * Visitor and page-view smoothed curves for a single shared time range.
*/ */
import { computed } from 'vue' import { computed, onMounted, onUnmounted, ref } from 'vue'
import { makeSeries } from './analytics/time.js' import { makeSeries } from './analytics/time.js'
import { CHART_H, CHART_W, buildChart, fmtY } from './analytics/chart.js' import { CHART_H, CHART_W, buildChart, fmtY } from './analytics/chart.js'
const DAY_REFRESH_MS = 15000
const props = defineProps({ const props = defineProps({
data: { type: Object, default: null }, data: { type: Object, default: null },
range: { type: String, required: true }, range: { type: String, required: true },
@@ -22,33 +24,52 @@ const allViews = computed(() => {
const visitSeries = computed(() => makeSeries(props.data?.site_visits, props.range)) const visitSeries = computed(() => makeSeries(props.data?.site_visits, props.range))
const viewSeries = computed(() => makeSeries(allViews.value, props.range)) const viewSeries = computed(() => makeSeries(allViews.value, props.range))
const unit = computed(() => (props.range === 'week' ? 'h' : 'day'))
const visitChart = computed(() => buildChart(visitSeries.value)) function freqLabel(unit) {
const viewChart = computed(() => buildChart(viewSeries.value)) return unit === '5min' ? '5 min' : unit === 'hour' ? 'hourly' : 'daily'
}
const now = ref(Date.now())
let refreshInterval = null
onMounted(() => {
refreshInterval = setInterval(() => { now.value = Date.now() }, DAY_REFRESH_MS)
})
onUnmounted(() => {
if (refreshInterval) clearInterval(refreshInterval)
})
const visitChart = computed(() => buildChart(visitSeries.value, now.value))
const viewChart = computed(() => buildChart(viewSeries.value, now.value))
</script> </script>
<template> <template>
<section v-for="c in [ <section v-for="c in [
{ ylabel: 'visitors', chart: visitChart, empty: 'no visits recorded yet' }, { ylabel: 'visits', chart: visitChart, empty: 'no visits recorded yet' },
{ ylabel: 'views', chart: viewChart, empty: 'no views recorded yet' }, { ylabel: 'views', chart: viewChart, empty: 'no views recorded yet' },
]" :key="c.ylabel"> ]" :key="c.ylabel">
<template v-if="c.chart"> <template v-if="c.chart">
<div class="chartwrap"> <div class="chartwrap">
<div class="plot"> <div class="plot">
<div class="plotarea"> <div class="plotarea">
<span class="yaxis-label">{{ c.ylabel }}/{{ unit }}</span> <span class="yaxis-label">{{ freqLabel(c.chart.unit) }} {{ c.ylabel }}</span>
<svg class="chart" :viewBox="`0 0 ${CHART_W} ${CHART_H}`" <svg class="chart" :viewBox="`0 0 ${CHART_W} ${CHART_H}`"
preserveAspectRatio="none" role="img" :aria-label="`${c.ylabel} per ${unit}`"> preserveAspectRatio="none" role="img" :aria-label="`${freqLabel(c.chart.unit)} ${c.ylabel}`">
<line v-for="g in c.chart.majors.slice(1)" :key="'j' + g.value" <line v-for="g in c.chart.majors.slice(1)" :key="'j' + g.value"
:x1="0" :x2="CHART_W" :y1="g.y" :y2="g.y" class="major" /> :x1="0" :x2="CHART_W" :y1="g.y" :y2="g.y" class="major" />
<template v-for="t in c.chart.xticks" :key="'t' + t.x"> <template v-for="t in c.chart.xticks" :key="'t' + t.x">
<line v-if="t.line" :x1="t.x" :x2="t.x" :y1="0" :y2="CHART_H" <line v-if="t.line" :x1="t.x" :x2="t.x" :y1="0" :y2="CHART_H"
class="minor vertical" /> class="minor vertical" />
</template> </template>
<template v-for="(s, i) in c.chart.series" :key="i"> <template v-if="c.chart.bars">
<path v-if="s.area" :d="s.area" class="area" /> <rect v-for="(b, i) in c.chart.bars" :key="'b' + i"
<path :d="s.line" class="line" :style="{ opacity: s.opacity }" /> :x="b.x" :y="b.y" :width="b.width" :height="b.height" class="bar" />
<path :d="c.chart.skyline" class="line" />
</template>
<template v-else>
<template v-for="(s, i) in c.chart.series" :key="i">
<path v-if="s.area" :d="s.area" class="area" />
<path :d="s.line" class="line" :style="{ opacity: s.opacity }" />
</template>
</template> </template>
<line :x1="0" :x2="CHART_W" :y1="CHART_H - 0.5" :y2="CHART_H - 0.5" <line :x1="0" :x2="CHART_W" :y1="CHART_H - 0.5" :y2="CHART_H - 0.5"
class="axis" /> class="axis" />
@@ -62,7 +83,7 @@ const viewChart = computed(() => buildChart(viewSeries.value))
</div> </div>
</div> </div>
</div> </div>
<div v-if="c.chart.series.length > 1" class="legend"> <div v-if="c.chart.series && c.chart.series.length > 1" class="legend">
<span v-for="(s, i) in c.chart.series" :key="i" :style="{ opacity: s.opacity }"> <span v-for="(s, i) in c.chart.series" :key="i" :style="{ opacity: s.opacity }">
● {{ s.label }} ● {{ s.label }}
</span> </span>
@@ -76,7 +97,7 @@ const viewChart = computed(() => buildChart(viewSeries.value))
/* The svg is stretched (preserveAspectRatio none), so all text lives in /* The svg is stretched (preserveAspectRatio none), so all text lives in
HTML overlays positioned by the same fractions the geometry uses. */ HTML overlays positioned by the same fractions the geometry uses. */
.chartwrap { .chartwrap {
padding-left: 2.2rem; /* y labels */ padding-left: 2.8rem; /* y labels */
} }
.plot { .plot {
@@ -103,8 +124,8 @@ const viewChart = computed(() => buildChart(viewSeries.value))
.ylab { .ylab {
position: absolute; position: absolute;
left: -2.2rem; left: -2.8rem;
width: 1.9rem; width: 2.6rem;
text-align: right; text-align: right;
transform: translateY(50%); transform: translateY(50%);
font-size: 0.7rem; font-size: 0.7rem;
@@ -154,6 +175,11 @@ const viewChart = computed(() => buildChart(viewSeries.value))
opacity: 0.15; opacity: 0.15;
} }
.chart .bar {
fill: var(--accent);
opacity: 0.15;
}
.chart .line { .chart .line {
fill: none; fill: none;
stroke: var(--accent); stroke: var(--accent);
@@ -176,10 +202,11 @@ const viewChart = computed(() => buildChart(viewSeries.value))
.yaxis-label { .yaxis-label {
position: absolute; position: absolute;
top: 50%; top: 50%;
left: -2.2rem; left: -2.8rem;
font-size: 0.7rem; font-size: 0.7rem;
color: var(--muted); color: var(--muted);
writing-mode: vertical-rl; writing-mode: vertical-rl;
white-space: nowrap;
transform: translateY(-50%) rotate(180deg); transform: translateY(-50%) rotate(180deg);
} }
frontend/src/analytics/chart.js
+89 -4
View File
@@ -5,7 +5,7 @@
* rates (hour on the week view, day on month+). * rates (hour on the week view, day on month+).
*/ */
import { DAY, HOUR, WEEK, mondayUTC } from './time.js' import { DAY, HOUR, MIN5, WEEK, mondayUTC } from './time.js'
export const CHART_W = 720 export const CHART_W = 720
export const CHART_H = 180 export const CHART_H = 180
@@ -206,9 +206,10 @@ export function spline(pts) {
} }
/** Build a full chart model from a series descriptor produced by time.js. */ /** Build a full chart model from a series descriptor produced by time.js. */
export function buildChart(input) { export function buildChart(input, now = Date.now()) {
if (!input || !input.series.length) return null if (!input || !input.series.length) return null
const { series, t0, t1, rate, binMinutes, unitMinutes } = input if (input.unit === '5min') return buildDayChart(input, now)
const { series, t0, t1, rate, binMinutes, unitMinutes, unit } = input
// Values are per-unit rates (hour on the week view, day on month+); the // Values are per-unit rates (hour on the week view, day on month+); the
// y max is derived from the *smoothed* curves so random single-bucket // y max is derived from the *smoothed* curves so random single-bucket
// spikes don't blow up the scale. Smoothing works on raw counts (its edge // spikes don't blow up the scale. Smoothing works on raw counts (its edge
@@ -283,7 +284,91 @@ export function buildChart(input) {
x: x(t), left: ((t - t0) / (t1 - t0)) * 100, x: x(t), left: ((t - t0) / (t1 - t0)) * 100,
label: fmtTick(t, t1 - t0), line: true, label: fmtTick(t, t1 - t0), line: true,
})) }))
return { max, majors, minors, series: drawn, xticks } return { max, majors, minors, series: drawn, xticks, unit }
}
/**
* Day view: 5-minute bars for the last 24 hours. Bars are drawn at raw
* counts; the skyline uses a projected full-bucket value for the still-open
* final bucket. The y scale is derived from the projected skyline maximum.
*/
export function buildDayChart(input, now = Date.now()) {
const { series, t0, t1 } = input
const points = series[0]?.points || []
const n = points.length
if (!n) return null
const bucketMs = (t1 - t0) / n
const bucketWidth = CHART_W / n
const gap = 0.2
const barWidth = Math.max(0.2, bucketWidth - gap)
const x = (i) => i * bucketWidth + gap / 2
const prevRaw = n > 1 ? points[n - 2].count : 0
const projected = points.map((p, i) => {
if (i !== n - 1) return p.count
const bucketStart = t0 + i * bucketMs
const elapsed = Math.max(1, Math.min(bucketMs, now - bucketStart))
// Blend the observed partial bucket with the previous full bucket:
// the longer the current bucket has run, the less we borrow from it.
const share = elapsed / bucketMs
return p.count + prevRaw * (1 - share)
})
const highest = Math.max(0, ...projected)
const { max, step, minor } = yScale(highest)
const y = (v) => PAD_TOP + (1 - Math.max(0, v) / max) * (CHART_H - PAD_TOP)
const bars = points.map((p, i) => {
const bx = x(i)
const by = y(p.count)
return {
x: bx,
y: by,
width: barWidth,
height: CHART_H - by,
raw: p.count,
projected: projected[i],
}
})
let skyline = ''
for (let i = 0; i < bars.length; i++) {
const b = bars[i]
const top = y(b.projected)
if (i === 0) {
skyline += `M${b.x},${top} H${b.x + b.width}`
} else {
skyline += ` V${top} H${b.x + b.width}`
}
}
const majors = []
const minors = []
const nMajor = Math.round(max / step)
for (let k = 0; k <= nMajor; k++) {
const v = k * step
majors.push({ value: v, y: y(v), bottom: (1 - PAD_TOP / CHART_H) * (v / max) * 100 })
}
if (minor) {
for (let v = minor; v < max; v += minor) {
if (v % step !== 0) minors.push({ y: y(v) })
}
}
const xticks = []
const tickStep = 3 * HOUR
const firstTick = Math.ceil(t0 / tickStep) * tickStep
for (let t = firstTick; t < t1; t += tickStep) {
if (t < t0) continue
const d = new Date(t)
xticks.push({
x: ((t - t0) / (t1 - t0)) * CHART_W,
left: ((t - t0) / (t1 - t0)) * 100,
label: `${String(d.getUTCHours()).padStart(2, '0')}:00`,
line: false,
})
}
return { bars, skyline: skyline.trim(), max, majors, minors, xticks, unit: '5min', series: [] }
} }
/** X ticks for year/all: Monday boundaries up to a quarter, UTC month /** X ticks for year/all: Monday boundaries up to a quarter, UTC month
frontend/src/analytics/format.js
+154 -9
View File
@@ -25,11 +25,44 @@ export const hostIP = (ip) => {
} }
} }
/** Copy the full IP to the clipboard, ignoring failures. */ function showCopiedFeedback(el) {
export async function copyIp(ip) { if (!el || typeof document === 'undefined') return
const popup = document.createElement('span')
popup.textContent = 'Copied!'
popup.className = 'copy-popup'
popup.style.cssText =
'position:absolute;bottom:calc(100% + 0.25rem);left:50%;' +
'transform:translateX(-50%);padding:0.15rem 0.4rem;' +
'background:var(--text, CanvasText);color:var(--bg, Canvas);' +
'border-radius:0.25rem;font-size:0.75rem;white-space:nowrap;' +
'pointer-events:none;z-index:10;'
el.classList.add('has-copy-popup')
el.appendChild(popup)
setTimeout(() => {
popup.remove()
el.classList.remove('has-copy-popup')
}, 1200)
}
/** Copy the full IP to the clipboard and show a brief "Copied!" popup. */
export async function copyIp(ip, event) {
if (!ip) return if (!ip) return
const el = event?.currentTarget
try { try {
await navigator.clipboard.writeText(ip) await navigator.clipboard.writeText(ip)
showCopiedFeedback(el)
} catch {
/* ignore */
}
}
/** Copy arbitrary text to the clipboard and show a brief "Copied!" popup. */
export async function copyList(text, event) {
if (!text) return
const el = event?.currentTarget
try {
await navigator.clipboard.writeText(text)
showCopiedFeedback(el)
} catch { } catch {
/* ignore */ /* ignore */
} }
@@ -140,6 +173,33 @@ export function formatWhenTooltip(ts) {
return new Date(ts).toISOString().replace('T', ' ').replace('Z', ' UTC') return new Date(ts).toISOString().replace('T', ' ').replace('Z', ' UTC')
} }
/** Full local timestamp for tooltips, e.g. "21 Aug 2026, 17:38:48". */
export function formatWhenLocal(ts) {
return new Date(ts).toLocaleString('en-ie', {
year: 'numeric',
month: 'short',
day: 'numeric',
hour: '2-digit',
minute: '2-digit',
second: '2-digit',
})
}
/** Preserve locale case with the region/country subtag upper-cased. */
export function formatLang(value) {
if (!value || value === '—') return value
const parts = value.split('-')
if (parts.length > 1) {
parts[parts.length - 1] = parts[parts.length - 1].toUpperCase()
}
return parts.join('-')
}
/** ISO 8601 UTC timestamp without subseconds, e.g. "2026-08-21T00:20:48Z". */
export function formatWhenIso(ts) {
return `${new Date(ts).toISOString().split('.')[0]}Z`
}
/** /**
* Format recent visits for display, newest first. Each step is a linked slug * Format recent visits for display, newest first. Each step is a linked slug
* pointing to its article; external referers/origins are shown as their * pointing to its article; external referers/origins are shown as their
@@ -241,8 +301,9 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
.sort((a, b) => totalHits(b) - totalHits(a) || b.lastStart - a.lastStart) .sort((a, b) => totalHits(b) - totalHits(a) || b.lastStart - a.lastStart)
.slice(0, 10) .slice(0, 10)
.map((g) => ({ .map((g) => ({
when: formatWhen(g.lastStart, now), lastSeen: formatWhen(g.lastStart, now),
whenTooltip: formatWhenTooltip(g.lastStart), lastSeenIso: formatWhenIso(g.lastStart),
lastSeenLocal: formatWhenLocal(g.lastStart),
pages: [...g.pages.entries()] pages: [...g.pages.entries()]
.sort((a, b) => b[1] - a[1]) .sort((a, b) => b[1] - a[1])
.map(([path, count]) => ({ .map(([path, count]) => ({
@@ -259,6 +320,85 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
})) }))
} }
/**
* Group abuse hits by IP (never by UA — scanners randomize theirs to
* masquerade as legitimate crawlers) and format each group as a row with
* the full paths probed, in access order. Flagged paths (the ones that
* triggered abuse classification) are lifted to the top, followed by
* other 404s, then document GETs from the abuser. UAs are shown raw,
* one per line, with their occurrence counts. Paths are shown verbatim
* (query string included), not resolved against the page tree.
*/
export function formatAbuseRows(abuse, now = Date.now()) {
const groups = new Map()
for (const a of abuse || []) {
const g = groups.get(a.ip) || {
ip: a.ip || '',
pathHits: [],
rawUas: [],
uaCounts: new Map(),
lastStart: 0,
}
const start = new Date(a.start).getTime()
if (start > g.lastStart) g.lastStart = start
g.pathHits.push({
path: a.path || '',
start,
flag: a.flag || false,
is_404: a.is_404 || false,
})
const ua = a.ua || '(no UA)'
g.rawUas.push(ua)
g.uaCounts.set(ua, (g.uaCounts.get(ua) || 0) + 1)
groups.set(a.ip, g)
}
const totalHits = (g) => g.pathHits.length
return [...groups.values()]
.sort((a, b) => totalHits(b) - totalHits(a) || b.lastStart - a.lastStart)
.slice(0, 10)
.map((g) => {
const pathCategory = (p) => (p.flag ? 0 : p.is_404 ? 1 : 2)
const paths = [...g.pathHits].sort(
(a, b) => pathCategory(a) - pathCategory(b) || a.start - b.start,
)
const uas = [...g.uaCounts.entries()]
.sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))
return {
lastSeen: formatWhen(g.lastStart, now),
lastSeenIso: formatWhenIso(g.lastStart),
lastSeenLocal: formatWhenLocal(g.lastStart),
paths: paths.map((p) => ({ path: p.path, flag: p.flag, is_404: p.is_404 })),
allPaths: paths.map((p) => p.path).join('\n'),
uas: uas.map(([ua, count]) => ({ ua, count })),
allUas: uas
.map(([ua, count]) => (count > 1 ? `${count}× ${ua}` : ua))
.join('\n'),
ip: g.ip,
ipDisplay: hostIP(g.ip) || g.ip || '—',
total: totalHits(g),
}
})
}
/**
* Reduce a reverse-DNS hostname to its right-most components that fit
* within ``limit`` characters. This keeps the meaningful main domain
* while avoiding absurdly long subdomains like ``xxx.yyy.zzz...provider.net``.
*/
function mainDomain(host, limit = 24) {
if (!host) return host
const labels = host.split('.').filter(Boolean)
if (!labels.length) return host
const parts = [labels.pop()]
while (labels.length) {
const next = labels[labels.length - 1]
const candidate = `${next}.${parts.join('.')}`
if (candidate.length > limit) break
parts.unshift(labels.pop())
}
return parts.join('.')
}
/** /**
* Format raw visit records as rows for a technical table. Returns objects * Format raw visit records as rows for a technical table. Returns objects
* with display strings; missing values become "—". ``trail`` starts with the * with display strings; missing values become "—". ``trail`` starts with the
@@ -268,21 +408,26 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
export function formatVisitRows(visits, pageTree, now = Date.now()) { export function formatVisitRows(visits, pageTree, now = Date.now()) {
const titles = buildTitleMap(pageTree) const titles = buildTitleMap(pageTree)
return [...(visits || [])].reverse().slice(0, 20).map((v) => { return [...(visits || [])].reverse().slice(0, 20).map((v) => {
const trail = [v.referer, v.entry, ...(v.trail || [])] const trail = [v.entry, ...(v.trail || [])]
.map((p) => stepOf(p, titles)) .map((p) => stepOf(p, titles))
.filter(Boolean) .filter(Boolean)
const utm = Object.entries(v.utm || {}) const utm = Object.entries(v.utm || {})
.map(([k, value]) => `${k}=${value}`) .map(([k, value]) => `${k}=${value}`)
.join(', ') .join(', ')
const dash = (s) => (s || '—') const dash = (s) => (s || '—')
const host = v.host || ''
const isHost = !!host
return { return {
when: formatWhen(v.start, now), lastSeen: formatWhen(v.start, now),
whenTooltip: formatWhenTooltip(v.start), lastSeenIso: formatWhenIso(v.start),
lastSeenLocal: formatWhenLocal(v.start),
langDisplay: formatLang(v.lang),
trail, trail,
refererStep: stepOf(v.referer, titles),
referer: dash(v.referer), referer: dash(v.referer),
ip: v.ip || '', ip: v.ip || '',
ipDisplay: v.host || hostIP(v.ip) || v.ip || '—', ipDisplay: isHost ? mainDomain(host) : hostIP(v.ip) || v.ip || '—',
host: dash(v.host), isHost,
lang: dash(v.lang), lang: dash(v.lang),
country: dash(v.country), country: dash(v.country),
city: dash(v.city), city: dash(v.city),
frontend/src/analytics/time.js
+30 -4
View File
@@ -13,6 +13,7 @@ export const DAY = 86400e3
export const WEEK = 7 * DAY export const WEEK = 7 * DAY
export const RANGES = { export const RANGES = {
day: { label: 'day', span: DAY, bucket: MIN5 },
week: { label: 'week' }, week: { label: 'week' },
month: { label: 'month', span: 30 * DAY, bucket: 6 * HOUR }, month: { label: 'month', span: 30 * DAY, bucket: 6 * HOUR },
year: { label: 'year', span: 365 * DAY, bucket: DAY }, year: { label: 'year', span: 365 * DAY, bucket: DAY },
@@ -117,11 +118,36 @@ export function rollingSeries(buckets, rangeKey) {
} }
} }
/** Dispatch to weekly or rolling series based on the selected range. */ /**
* Day view: raw 5-minute bucket counts for the current 24-hour window.
* No smoothing or rate conversion is applied; counts are used as-is.
*/
export function daySeries(buckets) {
const raw = rawTimes(buckets)
const now = Date.now()
const { span, bucket } = RANGES.day
const t1 = Math.floor(now / bucket) * bucket + bucket
const t0 = t1 - span
const points = []
for (let t = t0; t < t1; t += bucket) {
points.push({ t, count: raw[t] || 0 })
}
return {
series: [{ points, label: '', opacity: 1, area: false }],
t0,
t1,
rate: 1,
binMinutes: bucket / 60e3,
unitMinutes: bucket / 60e3,
unit: '5min',
}
}
/** Dispatch to daily, weekly or rolling series based on the selected range. */
export function makeSeries(buckets, rangeKey) { export function makeSeries(buckets, rangeKey) {
return rangeKey === 'week' if (rangeKey === 'day') return daySeries(buckets)
? weeklySeries(buckets) if (rangeKey === 'week') return weeklySeries(buckets)
: rollingSeries(buckets, rangeKey) return rollingSeries(buckets, rangeKey)
} }
/** /**
frontend/src/analytics/transitions.js
+72 -32
View File
@@ -23,22 +23,23 @@ export const TNODE_R = 34 // node circles hold the slug and the view count
export const EXT_R = 34 // external referer/exit nodes use the same full size export const EXT_R = 34 // external referer/exit nodes use the same full size
// Edge width (half-width of the thin middle) grows logarithmically with // Edge width (half-width of the thin middle) grows logarithmically with
// the count, anchored so a single recorded transition renders as a ~1 px // the count. The constants are scaled down by ~10× so busy ranges (day,
// line. There is no cap — growth is slow enough that even very hot // year) do not overwhelm the graph with fat connectors. A single recorded
// connections stay reasonable. Connections carrying less than // transition still renders as a faint ~0.4 px line. Connections carrying
// PRUNE_FRACTION of the total traffic are not drawn at all (this also // less than PRUNE_FRACTION of the total traffic are not drawn at all (this
// keeps the number of drawn connections under ~100). // also keeps the graph under ~100 connections).
const WMID_MIN = 0.5 const WMID_MIN = 0.2
const WIDTH_GROWTH = 1.5 const WIDTH_GROWTH = 0.15
const PRUNE_FRACTION = 0.01 const PRUNE_FRACTION = 0.01
// Beads: each edge direction emits beads at count * BEAD_RATE beads per // Beads: each edge direction emits beads at count * BEAD_RATE beads per
// second (linear in the count). The component simulates every bead // second (linear in the count). The rate is reduced ~10× across all time
// independently in JS at BEAD_SPEED along the edge, with no limit on // scales to keep the animation lightweight. The component simulates every
// bead independently in JS at BEAD_SPEED along the edge, with no limit on
// beads in flight. // beads in flight.
export const BEAD_SPEED = 180 // svg units per second export const BEAD_SPEED = 180 // svg units per second
export const BEAD_R = 2.2 export const BEAD_R = 2.2
const BEAD_RATE = 0.12 // beads per second per recorded transition const BEAD_RATE = 0.012 // beads per second per recorded transition
const FLOW_OFFSET = 3 // lane offset to the right of the travel direction const FLOW_OFFSET = 3 // lane offset to the right of the travel direction
const MAX_EXT_IN = 8 // referer nodes in the top row const MAX_EXT_IN = 8 // referer nodes in the top row
@@ -84,10 +85,15 @@ function collectInternalTransitions(transitions) {
return internal return internal
} }
/** Short display label for an external origin (protocol stripped). */ /** Domain-only label for an external origin (path removed). */
function extLabel(ext) { function extLabel(ext) {
const s = ext.replace(/^https?:\/\//, '') try {
return s.length > 11 ? `${s.slice(0, 10)}…` : s const host = new URL(ext).hostname
return host.length > 25 ? `${host.slice(0, 24)}…` : host
} catch {
const s = ext.replace(/^https?:\/\//, '').split('/')[0]
return s.length > 25 ? `${s.slice(0, 24)}…` : s
}
} }
/** /**
@@ -175,8 +181,27 @@ function layoutAngles(root, unit, weight) {
} }
} }
/** Compute median reading time per article in whole minutes. */
function buildReadMinutes(visits) {
const times = {}
for (const v of visits || []) {
for (const [path, sec] of Object.entries(v.read || {})) {
;(times[path] || (times[path] = [])).push(sec)
}
}
const minutes = {}
for (const [path, arr] of Object.entries(times)) {
arr.sort((a, b) => a - b)
const mid = Math.floor(arr.length / 2)
const median =
arr.length % 2 ? arr[mid] : (arr[mid - 1] + arr[mid]) / 2
minutes[path] = Math.max(1, Math.round(median / 60))
}
return minutes
}
/** Compute radial positions, view counts and labels for each node. */ /** Compute radial positions, view counts and labels for each node. */
function positionNodes(nodes, maxDepth, unit, viewsData, titles) { function positionNodes(nodes, maxDepth, unit, viewsData, titles, readMinutes) {
// Constant radial gap between rings, equal to the arc spacing of nodes // Constant radial gap between rings, equal to the arc spacing of nodes
// along a ring: leaf arc = unit * GAP, so GAP scales up with `unit` on // along a ring: leaf arc = unit * GAP, so GAP scales up with `unit` on
// sparse trees (where closing the circle forces wider arcs) and with // sparse trees (where closing the circle forces wider arcs) and with
@@ -196,10 +221,14 @@ function positionNodes(nodes, maxDepth, unit, viewsData, titles) {
n.x = Math.cos(n.angle) * r n.x = Math.cos(n.angle) * r
n.y = Math.sin(n.angle) * r n.y = Math.sin(n.angle) * r
n.views = viewCount(n.path) n.views = viewCount(n.path)
n.readMin = readMinutes[n.path] || 0
// Slug inside the circle; full title goes on the link title attribute. // Slug inside the circle; full title goes on the link title attribute.
const slug = n.path === '/' ? '🏠' : n.path.split('/').pop() const slug = n.path === '/' ? '🏠' : n.path.split('/').pop()
n.label = slug.length > 11 ? `${slug.slice(0, 10)}…` : slug n.label = slug.length > 16 ? `${slug.slice(0, 15)}…` : slug
n.title = titles.get(n.path) || '' n.title = titles.get(n.path) || ''
// Category (non-leaf) pages with no views in this window are left
// blank to keep the layout, but their circle/label is not drawn.
n.hidden = n.children.length > 0 && n.views === 0
} }
return { radius, GAP } return { radius, GAP }
@@ -279,7 +308,7 @@ const fmtPt = (p) => `${p[0].toFixed(2)} ${p[1].toFixed(2)}`
* `wMid` is the half-width of the thin middle (already strength-scaled by * `wMid` is the half-width of the thin middle (already strength-scaled by
* the caller); `ra`/`rb` are the radii of the node circles each end wraps. * the caller); `ra`/`rb` are the radii of the node circles each end wraps.
*/ */
function buildRibbon(a, b, ab, ba, wMid, ra = TNODE_R, rb = TNODE_R) { function buildRibbon(a, b, ab, ba, wMid, ra = TNODE_R, rb = TNODE_R, external = false) {
const count = ab + ba const count = ab + ba
const len = Math.hypot(b.x - a.x, b.y - a.y) || 1 const len = Math.hypot(b.x - a.x, b.y - a.y) || 1
const ux = (b.x - a.x) / len const ux = (b.x - a.x) / len
@@ -387,6 +416,7 @@ function buildRibbon(a, b, ab, ba, wMid, ra = TNODE_R, rb = TNODE_R) {
return { return {
d, d,
title: `${a.path} ↔ ${b.path}: ${count} (${ab} / ${ba})`, title: `${a.path} ↔ ${b.path}: ${count} (${ab} / ${ba})`,
external,
} }
} }
@@ -401,7 +431,7 @@ function buildRibbon(a, b, ab, ba, wMid, ra = TNODE_R, rb = TNODE_R) {
* edge run on parallel lanes instead of colliding. The component turns * edge run on parallel lanes instead of colliding. The component turns
* these into independently simulated beads. * these into independently simulated beads.
*/ */
function buildFlows(a, b, ra, rb, ab, ba) { function buildFlows(a, b, ra, rb, ab, ba, visualScale = 1) {
const len = Math.hypot(b.x - a.x, b.y - a.y) || 1 const len = Math.hypot(b.x - a.x, b.y - a.y) || 1
const ux = (b.x - a.x) / len const ux = (b.x - a.x) / len
const uy = (b.y - a.y) / len const uy = (b.y - a.y) / len
@@ -422,7 +452,7 @@ function buildFlows(a, b, ra, rb, ab, ba) {
x2: a.x + toT * ux + s * rx, x2: a.x + toT * ux + s * rx,
y2: a.y + toT * uy + s * ry, y2: a.y + toT * uy + s * ry,
len: span, len: span,
interval: 1 / (count * BEAD_RATE), interval: 1 / (count * BEAD_RATE * visualScale),
} }
} }
const flows = [] const flows = []
@@ -437,14 +467,17 @@ function buildFlows(a, b, ra, rb, ab, ba) {
* Absolute on purpose — cool routes stay visible regardless of how hot * Absolute on purpose — cool routes stay visible regardless of how hot
* the hottest connection is. * the hottest connection is.
*/ */
const scaledWidth = (count) => WMID_MIN + WIDTH_GROWTH * Math.log1p(count - 1) const scaledWidth = (count) => {
if (count <= 0) return 0
return WMID_MIN + WIDTH_GROWTH * Math.log1p(count - 1)
}
/** /**
* Build ribbon edges and bead flows for every aggregated page-to-page * Build ribbon edges and bead flows for every aggregated page-to-page
* pair. Pairs carrying less than PRUNE_FRACTION of the total internal * pair. Pairs carrying less than PRUNE_FRACTION of the total internal
* traffic are pruned (this naturally bounds the graph to ~100 edges). * traffic are pruned (this naturally bounds the graph to ~100 edges).
*/ */
function buildInternalEdges(pairs, byPath) { function buildInternalEdges(pairs, byPath, visualScale = 1) {
let total = 0 let total = 0
for (const [, [ab, ba]] of pairs) total += ab + ba for (const [, [ab, ba]] of pairs) total += ab + ba
const minCount = total * PRUNE_FRACTION const minCount = total * PRUNE_FRACTION
@@ -456,8 +489,10 @@ function buildInternalEdges(pairs, byPath) {
const [pf, pt] = k.split(' ') const [pf, pt] = k.split(' ')
const a = byPath.get(pf) const a = byPath.get(pf)
const b = byPath.get(pt) const b = byPath.get(pt)
edges.push(buildRibbon(a, b, ab, ba, scaledWidth(ab + ba))) const wMid = scaledWidth((ab + ba) * visualScale)
flows.push(...buildFlows(a, b, TNODE_R, TNODE_R, ab, ba)) if (wMid <= 0) continue
edges.push(buildRibbon(a, b, ab, ba, wMid))
flows.push(...buildFlows(a, b, TNODE_R, TNODE_R, ab, ba, visualScale))
} }
return { edges, flows } return { edges, flows }
} }
@@ -507,7 +542,7 @@ export function filterViewsByRange(views, t0, t1) {
* Widths and pruning use the same log scale and traffic-share rule as * Widths and pruning use the same log scale and traffic-share rule as
* internal connections. * internal connections.
*/ */
function buildExternal(external, byPath, radius, innerBounds) { function buildExternal(external, byPath, radius, innerBounds, visualScale = 1) {
const extNodes = [] const extNodes = []
const edges = [] const edges = []
const flows = [] const flows = []
@@ -517,7 +552,7 @@ function buildExternal(external, byPath, radius, innerBounds) {
const live = external.filter((p) => byPath.has(p.page)) const live = external.filter((p) => byPath.has(p.page))
if (!live.length) return { extNodes, edges, flows } if (!live.length) return { extNodes, edges, flows }
const width = scaledWidth const width = (count) => scaledWidth(count * visualScale)
const overlaps = (x, y, r) => const overlaps = (x, y, r) =>
[...byPath.values(), ...extNodes].some( [...byPath.values(), ...extNodes].some(
@@ -547,8 +582,10 @@ function buildExternal(external, byPath, radius, innerBounds) {
extNodes.push(xn) extNodes.push(xn)
for (const p of ps) { for (const p of ps) {
const page = byPath.get(p.page) const page = byPath.get(p.page)
edges.push(buildRibbon(xn, page, p.in, 0, width(p.in), EXT_R, TNODE_R)) const wMid = width(p.in)
flows.push(...buildFlows(xn, page, EXT_R, TNODE_R, p.in, 0)) if (wMid <= 0) continue
edges.push(buildRibbon(xn, page, p.in, 0, wMid, EXT_R, TNODE_R, true))
flows.push(...buildFlows(xn, page, EXT_R, TNODE_R, p.in, 0, visualScale))
} }
}) })
} }
@@ -593,8 +630,10 @@ function buildExternal(external, byPath, radius, innerBounds) {
extNodes.push(xn) extNodes.push(xn)
} }
xn.count += p.out xn.count += p.out
edges.push(buildRibbon(page, xn, p.out, 0, width(p.out), TNODE_R, EXT_R)) const wMid = width(p.out)
flows.push(...buildFlows(page, xn, TNODE_R, EXT_R, p.out, 0)) if (wMid <= 0) continue
edges.push(buildRibbon(page, xn, p.out, 0, wMid, TNODE_R, EXT_R, true))
flows.push(...buildFlows(page, xn, TNODE_R, EXT_R, p.out, 0, visualScale))
} }
return { extNodes, edges, flows } return { extNodes, edges, flows }
@@ -605,11 +644,12 @@ function buildExternal(external, byPath, radius, innerBounds) {
* Returns { nodes, edges, flows, extNodes, arcs, bounds } or null when * Returns { nodes, edges, flows, extNodes, arcs, bounds } or null when
* there is nothing to show. * there is nothing to show.
*/ */
export function buildTransitionGraph(data, pageTree) { export function buildTransitionGraph(data, pageTree, visits = [], visualScale = 1) {
const internal = collectInternalTransitions(data?.transitions) const internal = collectInternalTransitions(data?.transitions)
const external = collectExternalPairs(data?.transitions) const external = collectExternalPairs(data?.transitions)
const navOrder = buildNavigationOrder(pageTree) const navOrder = buildNavigationOrder(pageTree)
const titles = buildTitleMap(pageTree) const titles = buildTitleMap(pageTree)
const readMinutes = buildReadMinutes(visits)
if (!internal.length && !navOrder.size) return null if (!internal.length && !navOrder.size) return null
@@ -619,10 +659,10 @@ export function buildTransitionGraph(data, pageTree) {
layoutAngles(root, unit, weightFn) layoutAngles(root, unit, weightFn)
const maxDepth = Math.max(1, ...nodes.map((n) => n.depth)) const maxDepth = Math.max(1, ...nodes.map((n) => n.depth))
const { radius } = positionNodes(nodes, maxDepth, unit, data?.views, titles) const { radius } = positionNodes(nodes, maxDepth, unit, data?.views, titles, readMinutes)
const arcs = buildFamilyArcs(nodes, radius) const arcs = buildFamilyArcs(nodes, radius)
const pairs = aggregatePairs(internal) const pairs = aggregatePairs(internal)
const { edges, flows } = buildInternalEdges(pairs, byPath) const { edges, flows } = buildInternalEdges(pairs, byPath, visualScale)
// Tight bounding box of the actual page nodes; family ring arcs can sweep // Tight bounding box of the actual page nodes; family ring arcs can sweep
// outside the node circle (e.g. a large arc between two siblings on the // outside the node circle (e.g. a large arc between two siblings on the
@@ -646,7 +686,7 @@ export function buildTransitionGraph(data, pageTree) {
bounds.y1 = Math.max(bounds.y1, b.y1) bounds.y1 = Math.max(bounds.y1, b.y1)
} }
const ext = buildExternal(external, byPath, radius, bounds) const ext = buildExternal(external, byPath, radius, bounds, visualScale)
for (const xn of ext.extNodes) { for (const xn of ext.extNodes) {
bounds.x0 = Math.min(bounds.x0, xn.x - xn.r - pad) bounds.x0 = Math.min(bounds.x0, xn.x - xn.r - pad)
bounds.y0 = Math.min(bounds.y0, xn.y - xn.r - pad) bounds.y0 = Math.min(bounds.y0, xn.y - xn.r - pad)
frontend/src/pagerite.js
+101 -15
View File
@@ -338,29 +338,107 @@ import "overlayscrollbars/overlayscrollbars.css";
} }
// --- Analytics pings --------------------------------------------------- // --- Analytics pings ---------------------------------------------------
// Fire-and-forget POST /_a {fr, to}: on the initial page load (starts the // Fire-and-forget POST /_a {fr, to, read}: on the initial page load
// visit — the server counts nothing from the document GET alone), for // (starts the visit — the server counts nothing from the document GET
// internal fetch-navigations and for external https exits. Excluded: // alone), for internal fetch-navigations, for external https exits, and
// back/forward (popstate never pings) and everything while we know the // on window close. ``read`` is the active time (ms) spent on ``fr``.
// user is an admin — but only when SSO is actually in use; with no auth // Reading time pauses after 1 minute of inactivity and resumes on the
// (dev/test) "admin" is everyone's state and nothing would be recorded — // next mouse/touch/scroll/keyboard event.
// or has the editor open (admin noise, not visits). The analytics page // Excluded: back/forward (popstate never pings), everything while the
// itself (/_a) is also excluded even though fetch-navigation treats it like // editor is open (body.editing — admin noise, not visits), and the
// a normal article. // analytics page itself (/_a), even though fetch-navigation treats it
// like a normal article.
// Admins (when SSO is actually in use — with no auth proxy "admin" is
// everyone's state) ping normally but with hide=1: the server then
// records nothing and scrubs any session the same browser accumulated
// before logging in, so admins never show up as visits or crawlers.
// See docs/analytics.md. // See docs/analytics.md.
function ping(to, fr = currentPath) { function ping(to, fr = currentPath, read = 0) {
if ((ssoAvailable && isAdmin) || document.body.classList.contains("editing") if (document.body.classList.contains("editing")) return;
|| to === "/_a" || fr === "/_a") return; if ((to && to === "/_a") || fr === "/_a") return;
const hide = ssoAvailable && isAdmin ? 1 : 0;
const body = JSON.stringify({
fr, to, hide,
read: Math.max(0, Math.round(read / 1000)),
});
try { try {
fetch("/_a", { fetch("/_a", {
method: "POST", method: "POST",
keepalive: true, keepalive: true,
headers: { "content-type": "application/json" }, headers: { "content-type": "application/json" },
body: JSON.stringify({ fr, to }), body,
}); });
} catch { /* analytics must never break navigation */ } } catch { /* analytics must never break navigation */ }
} }
// Active reading time for the current page. The clock stops after 1 minute
// without activity and restarts on the next mouse/touch/scroll/keyboard
// event.
const INACTIVE_MS = 60_000;
let readStart = performance.now();
let readElapsed = 0;
let reading = true;
let readInactivityTimer = null;
let closePingedFor = null;
function markReadActivity() {
if (!reading) {
reading = true;
readStart = performance.now();
}
clearTimeout(readInactivityTimer);
readInactivityTimer = setTimeout(() => {
if (reading) {
readElapsed += performance.now() - readStart;
reading = false;
}
}, INACTIVE_MS);
}
function takeReadTime() {
if (reading) {
readElapsed += performance.now() - readStart;
readStart = performance.now();
}
const ms = Math.max(0, Math.round(readElapsed));
readElapsed = 0;
return ms;
}
function resetReadTime() {
readElapsed = 0;
reading = true;
readStart = performance.now();
clearTimeout(readInactivityTimer);
}
function sendClosePing() {
if (closePingedFor === currentPath) return;
const read = Math.max(0, Math.round(takeReadTime() / 1000));
if (read <= 0) return;
const hide = ssoAvailable && isAdmin ? 1 : 0;
const body = JSON.stringify({ fr: currentPath, hide, read });
const blob = new Blob([body], { type: "application/json" });
try {
if (navigator.sendBeacon) {
navigator.sendBeacon("/_a", blob);
} else {
fetch("/_a", {
method: "POST",
keepalive: true,
headers: { "content-type": "application/json" },
body,
});
}
} catch { /* analytics must never break navigation */ }
closePingedFor = currentPath;
}
for (const ev of ["mousemove", "mousedown", "touchstart", "touchmove", "scroll", "keydown"]) {
addEventListener(ev, markReadActivity, { passive: true });
}
addEventListener("pagehide", sendClosePing);
// The initial page load pings too — it is what starts the visit and // The initial page load pings too — it is what starts the visit and
// counts the entry page view (the document GET alone records nothing). // counts the entry page view (the document GET alone records nothing).
// Sent once per load, after the auth probes so the admin gate applies; // Sent once per load, after the auth probes so the admin gate applies;
@@ -538,7 +616,10 @@ import "overlayscrollbars/overlayscrollbars.css";
if (url.origin !== location.origin) { if (url.origin !== location.origin) {
// External link: the browser navigates; record the full https URL so // External link: the browser navigates; record the full https URL so
// different links to the same domain stay distinct in analytics. // different links to the same domain stay distinct in analytics.
if (url.protocol === "https:") ping(url.href); if (url.protocol === "https:") {
closePingedFor = currentPath;
ping(url.href, currentPath, takeReadTime());
}
return; return;
} }
// Same-page anchor links (footnotes etc.): let the browser handle them // Same-page anchor links (footnotes etc.): let the browser handle them
@@ -550,7 +631,12 @@ import "overlayscrollbars/overlayscrollbars.css";
ev.preventDefault(); ev.preventDefault();
// Capture the source now: load() updates currentPath before pinging. // Capture the source now: load() updates currentPath before pinging.
const from = currentPath; const from = currentPath;
load(url).then((ok) => { if (ok) ping(url.pathname, from); }); load(url).then((ok) => {
if (!ok) return;
closePingedFor = null;
ping(url.pathname, from, takeReadTime());
resetReadTime();
});
}); });
addEventListener("popstate", () => { addEventListener("popstate", () => {
pagerite/analytics.py
+233 -13
View File
@@ -5,10 +5,14 @@ ping on page load starts a visit, later pings extend it, and pings with no
known session start a fresh one (missing data, not dropped). The document known session start a fresh one (missing data, not dropped). The document
GET handler stashes the entry referer (external https origin) and any GET handler stashes the entry referer (external https origin) and any
utm_* query parameters in in-memory IP tables, consumed when the ping utm_* query parameters in in-memory IP tables, consumed when the ping
starts the visit; nothing is counted without a ping (bots and admin starts the visit; nothing is counted without a ping (bots stay invisible).
browsing stay invisible). The session map is in-memory only. The visitor Admin clients ping with ``hide=1``, which records nothing and removes any
IP and, when available, its reverse-DNS host name are stored on the visit visit the session accumulated before logging in. Scanner telltale 404s
record itself. (dotpaths, *.php) classify the source IP as abuse; its hits — including
earlier crawler hits — are moved to the abuse list, which the viewer
groups by IP with full request paths. The session map is in-memory only.
The visitor IP and, when available, its reverse-DNS host name are stored
on the visit record itself.
Data is a msgspec Struct JSON-dumped to its own file (not the kanta db), Data is a msgspec Struct JSON-dumped to its own file (not the kanta db),
rewritten atomically on every recorded event. rewritten atomically on every recorded event.
@@ -41,7 +45,10 @@ def _compact_user_agent(ua: str) -> str:
dev = r.device.family if r.device else None dev = r.device.family if r.device else None
if browser in (None, "Other") and os_name in (None, "Other"): if browser in (None, "Other") and os_name in (None, "Other"):
return ua return ua
browser = browser if browser and browser != "Other" else "" if browser and browser != "Other":
browser = browser.split()[0]
else:
browser = ""
os_name = os_name if os_name and os_name != "Other" else "" os_name = os_name if os_name and os_name != "Other" else ""
if dev in (None, "Other") or dev == browser: if dev in (None, "Other") or dev == browser:
dev = "" dev = ""
@@ -79,6 +86,8 @@ class Visit(msgspec.Struct, omit_defaults=True):
ua_pretty: str = "" ua_pretty: str = ""
#: UTM query parameters from the landing URL, keyed by parameter name. #: UTM query parameters from the landing URL, keyed by parameter name.
utm: dict[str, str] = {} utm: dict[str, str] = {}
#: Active reading time per path (seconds), keyed by path.
read: dict[str, int] = {}
class CrawlerHit(msgspec.Struct, omit_defaults=True): class CrawlerHit(msgspec.Struct, omit_defaults=True):
@@ -96,6 +105,29 @@ class CrawlerHit(msgspec.Struct, omit_defaults=True):
query: str = "" query: str = ""
class AbuseHit(msgspec.Struct, omit_defaults=True):
"""A request from an IP classified as a scanner/abuser.
Unlike crawler hits the full request path (query string included) is
kept: the interesting part is exactly which paths were probed.
``flag`` marks the path that triggered classification; ``is_404``
distinguishes 404 responses from document GETs made by the abuser.
"""
start: datetime
#: Full request path including the query string (e.g. "/.env?x=1").
path: str
ip: str = ""
ua: str = ""
#: Compact display form of ``ua`` when parsable.
ua_pretty: str = ""
#: True when this path triggered abuse classification (telltale path
#: or the 404 that crossed the threshold).
flag: bool = False
#: True for 404 responses; false for document GETs from the abuser.
is_404: bool = False
class Analytics(msgspec.Struct, omit_defaults=True): class Analytics(msgspec.Struct, omit_defaults=True):
"""Root of the analytics JSON file. Append-only by design: old data is """Root of the analytics JSON file. Append-only by design: old data is
dropped by deleting list entries / bucket keys.""" dropped by deleting list entries / bucket keys."""
@@ -103,6 +135,10 @@ class Analytics(msgspec.Struct, omit_defaults=True):
visits: list[Visit] = [] visits: list[Visit] = []
#: Document GETs that never produced a ping, treated as crawler/bot hits. #: Document GETs that never produced a ping, treated as crawler/bot hits.
crawlers: list[CrawlerHit] = [] crawlers: list[CrawlerHit] = []
#: Requests from abusive IPs (see AbuseHit), grouped by IP in the viewer.
abuse: list[AbuseHit] = []
#: IPs classified as scanners/abusers (keys; values always True).
abuse_ips: dict[str, bool] = {}
#: Page transitions per 5-minute bucket (sparse): #: Page transitions per 5-minute bucket (sparse):
#: from -> to -> bucket ISO -> count. ``from`` is the referer origin or #: from -> to -> bucket ISO -> count. ``from`` is the referer origin or
#: "(direct)" for initial loads, a page path for pings. #: "(direct)" for initial loads, a page path for pings.
@@ -186,6 +222,19 @@ def _utm_tags(query: str) -> dict[str, str]:
_CRAWLER_TIMEOUT = timedelta(seconds=10) _CRAWLER_TIMEOUT = timedelta(seconds=10)
#: Plain-404 count per IP that classifies it as abuse even without a
#: telltale path hit.
_ABUSE_404_THRESHOLD = 10
#: Paths that instantly classify an IP as abuse when they 404: any segment
#: starting with a dot ("/.env", "/.git/config") or ending in ".php".
_ABUSE_PATH = re.compile(r"(^|/)\.|\.php$", re.IGNORECASE)
def _is_abuse_path(path: str) -> bool:
"""Telltale scanner path: dot segment or *.php."""
return bool(_ABUSE_PATH.search(path.split("?")[0]))
class Store: class Store:
"""In-memory analytics data plus the (IP, UA) -> visit session map.""" """In-memory analytics data plus the (IP, UA) -> visit session map."""
@@ -212,6 +261,9 @@ class Store:
#: Document GETs that have not yet been matched by a ping. Kept #: Document GETs that have not yet been matched by a ping. Kept
#: in RAM only; expired entries are written to ``data.crawlers``. #: in RAM only; expired entries are written to ``data.crawlers``.
self.pending_crawlers: list[CrawlerHit] = [] self.pending_crawlers: list[CrawlerHit] = []
#: ip -> number of plain (non-telltale) 404s seen, in RAM only;
#: reaching ``_ABUSE_404_THRESHOLD`` classifies the IP as abuse.
self.not_found_counts: dict[str, int] = {}
#: Callables to notify when persisted data changes. Registered by the #: Callables to notify when persisted data changes. Registered by the
#: analytics WebSocket broadcaster. #: analytics WebSocket broadcaster.
self._on_change: list[Callable[[], None]] = [] self._on_change: list[Callable[[], None]] = []
@@ -267,6 +319,123 @@ class Store:
buckets = self.data.transitions.setdefault(fr, {}).setdefault(to, {}) buckets = self.data.transitions.setdefault(fr, {}).setdefault(to, {})
self._count(buckets, _bucket(now)) self._count(buckets, _bucket(now))
def _uncount(self, table: dict[str, int], key: str) -> None:
"""Reverse one ``_count``: decrement and drop empty keys."""
if key in table:
table[key] -= 1
if table[key] <= 0:
del table[key]
def _remove_visit(self, index: int) -> None:
"""Delete a visit and reverse the counts its creation recorded.
Used when a known visitor turns out to be an admin (hide=1 ping):
the session is scrubbed from the stats. Views/transitions logged
by later pings inside the visit lack per-event timestamps and are
left as-is.
"""
visit = self.data.visits[index]
bucket = _bucket(visit.start)
self._uncount(self.data.site_visits, bucket)
views = self.data.views.get(visit.entry)
if views is not None:
self._uncount(views, bucket)
if not views:
del self.data.views[visit.entry]
fr_map = self.data.transitions.get(visit.referer or "(direct)")
if fr_map is not None:
buckets = fr_map.get(visit.entry)
if buckets is not None:
self._uncount(buckets, bucket)
if not buckets:
del fr_map[visit.entry]
if not fr_map:
del self.data.transitions[visit.referer or "(direct)"]
del self.data.visits[index]
# Sessions store list indices; shift the ones past the removed visit.
for key, i in list(self.sessions.items()):
if i > index:
self.sessions[key] = i - 1
def _abuse_hit(
self,
ip: str,
ua: str,
path: str,
start: datetime | None = None,
*,
flag: bool = False,
is_404: bool = False,
) -> None:
"""Append one abuse hit with the full request path."""
self.data.abuse.append(
AbuseHit(
start=start or datetime.now(UTC),
path=path,
ip=ip,
ua=ua,
ua_pretty=_compact_user_agent(ua),
flag=flag,
is_404=is_404,
)
)
def classify_abuse(
self,
ip: str,
ua: str,
path: str,
*,
flag: bool = False,
is_404: bool = False,
) -> None:
"""Classify an IP as a scanner/abuser and record the triggering hit.
All earlier crawler hits from the same IP (persisted and pending)
are moved to the abuse list — a random-UA scanner must not pollute
the crawler stats of the legitimate bots it impersonates.
"""
if ip not in self.data.abuse_ips:
self.data.abuse_ips[ip] = True
moved = [h for h in self.data.crawlers if h.ip == ip]
if moved:
self.data.crawlers = [h for h in self.data.crawlers if h.ip != ip]
for h in moved:
self._abuse_hit(
h.ip, h.ua,
h.entry + (f"?{h.query}" if h.query else ""),
start=h.start,
)
pending = [h for h in self.pending_crawlers if h.ip == ip]
if pending:
self.pending_crawlers = [h for h in self.pending_crawlers if h.ip != ip]
for h in pending:
self._abuse_hit(
h.ip, h.ua,
h.entry + (f"?{h.query}" if h.query else ""),
start=h.start,
)
self._abuse_hit(ip, ua, path, flag=flag, is_404=is_404)
self._save()
def track_404(self, ip: str, ua: str, path: str) -> None:
"""Record a 404 response for ``path`` (full path, query included).
A telltale path (dot segment or *.php) classifies the IP as abuse
immediately; enough plain 404s from one IP do too. Hits from
already-classified IPs go straight to the abuse list.
"""
if ip in self.data.abuse_ips:
self._abuse_hit(ip, ua, path, flag=_is_abuse_path(path), is_404=True)
self._save()
return
if _is_abuse_path(path):
self.classify_abuse(ip, ua, path, flag=True, is_404=True)
return
self.not_found_counts[ip] = self.not_found_counts.get(ip, 0) + 1
if self.not_found_counts[ip] >= _ABUSE_404_THRESHOLD:
self.classify_abuse(ip, ua, path, flag=True, is_404=True)
def _new_visit( def _new_visit(
self, self,
entry: str, entry: str,
@@ -328,8 +497,7 @@ class Store:
own_origin: str, own_origin: str,
ip: str, ip: str,
ua: str, ua: str,
entry: str, full_path: str,
query: str = "",
) -> None: ) -> None:
"""Stash the entry referer/UTM tags and queue a pending crawler hit. """Stash the entry referer/UTM tags and queue a pending crawler hit.
@@ -343,7 +511,17 @@ class Store:
Every document GET is also queued as a pending crawler hit. If a ping Every document GET is also queued as a pending crawler hit. If a ping
from the same (IP, UA) pair arrives within ``_CRAWLER_TIMEOUT``, the from the same (IP, UA) pair arrives within ``_CRAWLER_TIMEOUT``, the
hit is discarded; otherwise it is flushed to ``data.crawlers``. hit is discarded; otherwise it is flushed to ``data.crawlers``.
GETs from IPs already classified as abuse are recorded as abuse hits
with the full request path (query string included).
""" """
entry = full_path.split("?")[0]
query = full_path.split("?", 1)[1] if "?" in full_path else ""
if ip in self.data.abuse_ips:
self._flush_crawlers()
self._abuse_hit(ip, ua, full_path, is_404=False, flag=False)
self._save()
return
now = datetime.now(UTC) now = datetime.now(UTC)
self._flush_crawlers(now) self._flush_crawlers(now)
if referer: if referer:
@@ -365,31 +543,73 @@ class Store:
) )
) )
def _add_read(self, ip: str, ua: str, path: str, seconds: int) -> None:
"""Add ``seconds`` of reading time for ``path`` to the current visit."""
if seconds <= 0:
return
index = self.sessions.get((ip, ua))
if index is None or index >= len(self.data.visits):
return
visit = self.data.visits[index]
visit.read[path] = visit.read.get(path, 0) + seconds
def ping( def ping(
self, self,
from_: str, from_: str,
to: str, to: str | None,
ip: str, ip: str,
ua: str, ua: str,
accept_language: str = "", accept_language: str = "",
hide: bool = False,
read: int = 0,
) -> int | None: ) -> int | None:
"""Record a client navigation ping ({from, to} from pagerite.js). """Record a client navigation ping ({from, to, read} from pagerite.js).
``to`` is an internal path ("/...") or an https URL for exit links; a
missing/empty ``to`` means the page is being closed and only the
``read`` time should be recorded. The transition is always counted when
``to`` is present; the trail only grows on first sight of a page within
the visit. ``read`` is the active time (seconds) spent on ``from_``.
``to`` is an internal path ("/...") or an https URL for exit links;
anything else is ignored. The transition is always counted; the trail
only grows on first sight of a page within the visit.
A ping with no known session starts a fresh visit, consuming the A ping with no known session starts a fresh visit, consuming the
referer and UTM tags stashed by the document GET if there are any. referer and UTM tags stashed by the document GET if there are any.
``hide`` is set by admin clients: the ping cancels pending crawler
hits as usual, and any existing visit for this (IP, UA) session is
removed from the stats (the admin browsed anonymously before logging
in). Nothing new is recorded.
Pings from IPs classified as abuse are ignored entirely.
Returns the index of the new visit when one is created, so callers Returns the index of the new visit when one is created, so callers
can enrich it later with non-blocking lookups (host, geoip country). can enrich it later with non-blocking lookups (host, geoip country).
""" """
self._flush_crawlers() self._flush_crawlers()
key = (ip, ua)
if hide:
# Admin ping: cancel pending crawler hits and scrub the session.
self.pending_crawlers = [
hit for hit in self.pending_crawlers if not (hit.ip == ip and hit.ua == ua)
]
index = self.sessions.pop(key, None)
if index is not None and index < len(self.data.visits):
self._remove_visit(index)
self._save()
return None
if ip in self.data.abuse_ips:
return None
# A real visitor ping cancels any pending crawler hits from this # A real visitor ping cancels any pending crawler hits from this
# (IP, UA) pair. # (IP, UA) pair.
self.pending_crawlers = [ self.pending_crawlers = [
hit for hit in self.pending_crawlers if not (hit.ip == ip and hit.ua == ua) hit for hit in self.pending_crawlers if not (hit.ip == ip and hit.ua == ua)
] ]
fr_path = _internal_path(from_) if from_ else ""
if fr_path and read > 0:
self._add_read(ip, ua, fr_path, read)
if not to:
if read > 0:
self._save()
return None
if to.startswith("/") and not to.startswith("//"): if to.startswith("/") and not to.startswith("//"):
target = _internal_path(to) or "" target = _internal_path(to) or ""
else: else:
@@ -398,7 +618,7 @@ class Store:
return None return None
key = (ip, ua) key = (ip, ua)
index = self.sessions.get(key) index = self.sessions.get(key)
fr = (_internal_path(from_) or "(direct)") if from_ else "(direct)" fr = fr_path or "(direct)"
if index is None or index >= len(self.data.visits): if index is None or index >= len(self.data.visits):
# No known session: the initial ping of a fresh page load (or # No known session: the initial ping of a fresh page load (or
# missing data after a server restart) — start a visit. # missing data after a server restart) — start a visit.
pagerite/app.py
+39 -6
View File
@@ -126,13 +126,21 @@ class GeoIP:
return "" return ""
def city(self, ip: str) -> str: def city(self, ip: str) -> str:
"""City name for ``ip``, or "" when unavailable.""" """City name for ``ip``, or "" when unavailable.
GeoIP sometimes appends district names in parentheses (e.g.
"Berlin (Bezirk Tempelhof-Schöneberg)"); those are stripped before
the value is stored.
"""
if not ip or self._reader is None: if not ip or self._reader is None:
return "" return ""
try: try:
rec = self._reader.get(ip) rec = self._reader.get(ip)
if rec: if rec:
return (rec.get("city") or {}).get("names", {}).get("en", "") city = (rec.get("city") or {}).get("names", {}).get("en", "")
if city:
city = re.sub(r"\s*\([^)]*\)", "", city).strip()
return city
except Exception: except Exception:
pass pass
return "" return ""
@@ -605,6 +613,12 @@ def _client_ip(request: Request) -> str:
return forwarded or (request.client.host if request.client else "") return forwarded or (request.client.host if request.client else "")
def _query_suffix(request: Request) -> str:
"""The request's query string as a "?..." suffix, or "" when absent."""
query = str(request.url.query)
return f"?{query}" if query else ""
@lru_cache(maxsize=4096) @lru_cache(maxsize=4096)
def _cached_ptr(ip: str) -> str: def _cached_ptr(ip: str) -> str:
"""Reverse-DNS lookup with in-RAM LRU cache. Returns the host name or "".""" """Reverse-DNS lookup with in-RAM LRU cache. Returns the host name or ""."""
@@ -683,7 +697,11 @@ class AnalyticsPing(BaseModel):
"""Navigation ping from pagerite.js (see docs/analytics.md).""" """Navigation ping from pagerite.js (see docs/analytics.md)."""
fr: str = "" fr: str = ""
to: str to: str | None = None
#: 1 from admin clients: scrub the session instead of recording it.
hide: int = 0
#: Active reading time on ``fr`` (ms), if any.
read: int = 0
@app.get("/_a", response_model=None) @app.get("/_a", response_model=None)
@@ -716,6 +734,8 @@ async def analytics_ping(ping: AnalyticsPing, request: Request) -> None:
ip, ip,
request.headers.get("user-agent", ""), request.headers.get("user-agent", ""),
request.headers.get("accept-language", ""), request.headers.get("accept-language", ""),
hide=bool(ping.hide),
read=ping.read,
) )
if index is not None: if index is not None:
asyncio.create_task(_enrich_visit(index, ip)) asyncio.create_task(_enrich_visit(index, ip))
@@ -725,7 +745,8 @@ def _track_entry(path: str, request: Request) -> None:
"""Stash the referer/UTM tags and queue a pending crawler hit for the GET. """Stash the referer/UTM tags and queue a pending crawler hit for the GET.
Nothing is counted on the GET itself — the client's /_a ping starts the Nothing is counted on the GET itself — the client's /_a ping starts the
visit, so bots and admin browsing never register as visits. visit, so bots never register as visits. (Admin clients ping too, but
with hide=1, which scrubs their session instead of recording it.)
The devserver's health probe (``GET /?from=devserver.py`` from The devserver's health probe (``GET /?from=devserver.py`` from
``127.0.0.1``) is ignored: it is not real traffic and would otherwise be ``127.0.0.1``) is ignored: it is not real traffic and would otherwise be
@@ -739,13 +760,13 @@ def _track_entry(path: str, request: Request) -> None:
): ):
return return
own_origin = f"https://{urlparse(str(request.base_url)).netloc}" own_origin = f"https://{urlparse(str(request.base_url)).netloc}"
full_path = f"{request.url.path}{_query_suffix(request)}"
analytics_store.track_entry( analytics_store.track_entry(
request.headers.get("referer", ""), request.headers.get("referer", ""),
own_origin, own_origin,
_client_ip(request), _client_ip(request),
request.headers.get("user-agent", ""), request.headers.get("user-agent", ""),
"/" if path == "" else f"/{path}", full_path,
str(request.url.query),
) )
@@ -968,6 +989,13 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
if path and _is_reserved(path): if path and _is_reserved(path):
# Invalid slug shape: not a content URL, let FastAPI return its # Invalid slug shape: not a content URL, let FastAPI return its
# built-in 404 instead of rendering an editable article page. # built-in 404 instead of rendering an editable article page.
# Scanner telltales (dotpaths like /.env, *.php) classify the IP
# as abuse in analytics.
analytics_store.track_404(
_client_ip(request),
request.headers.get("user-agent", ""),
f"/{path}{_query_suffix(request)}",
)
raise HTTPException(404) raise HTTPException(404)
chain = resolve(data.menu, path) chain = resolve(data.menu, path)
node = chain[-1] if chain else None node = chain[-1] if chain else None
@@ -1011,5 +1039,10 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
if item.published: if item.published:
return RedirectResponse(f"/{slug}") return RedirectResponse(f"/{slug}")
if _is_trackable_path(path): if _is_trackable_path(path):
analytics_store.track_404(
_client_ip(request),
request.headers.get("user-agent", ""),
f"/{path}{_query_suffix(request)}",
)
_track_entry(path, request) _track_entry(path, request)
return HTMLResponse(views.render_not_found(data.menu, path, data.brand, data.custom_css, data.theme, data.favicon, data.brand_html), 404) return HTMLResponse(views.render_not_found(data.menu, path, data.brand, data.custom_css, data.theme, data.favicon, data.brand_html), 404)