LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

analytics: shared Client struct, grouped abuse paths, unified visitor cell

Browse Source
This commit is contained in:
LeoVasanko
2026-08-21 23:16:15 +00:00
parent 51a6a16221
commit fdb3e42d6f
7 changed files with 614 additions and 422 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/analytics.md
+74 -66
View File
@@ -5,8 +5,9 @@ Struct dumped to disk — separate from the kanta content database, path from
`PAGERITE_ANALYTICS` (default: the database path with `.kantadb` replaced by
`.analytics.json`, e.g. `pagerite.analytics.json`).
- `pagerite/analytics.py` — data model (`Analytics`, `Visit`) and the `Store`
(in-memory data + session map, atomic JSON persistence).
- `pagerite/analytics.py` — data model (`Analytics`, `Client`, `Visit`,
`CrawlerHit`, `AbuseHit`) and the `Store` (in-memory data + session map,
atomic JSON persistence).
- `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`),
the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics`
(admin-gated like every `/_api` endpoint).
@@ -42,40 +43,45 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
editor open (`body.editing`). Admin noise, not visits.
- **Admins**: when SSO is in use and the session is known to be an admin,
the client still pings but adds `hide=1`. The server then records
nothing — and if the same (IP, UA) session already had a visit from
before logging in, that visit is removed from the JSON along with the
counts recorded when it was created (site visit, entry view, entry
transition). Views/transitions logged by later pings inside such a visit
lack per-event timestamps and are left as-is. With no auth proxy
(dev/test) "admin" is everyone's state, so `hide` stays 0 and everything
is recorded.
nothing — and if the same client session already had a visit from before
logging in, that visit is removed from the JSON along with the counts
recorded when it was created (site visit, entry view, entry transition).
Views/transitions logged by later pings inside such a visit lack
per-event timestamps and are left as-is. With no auth proxy (dev/test)
"admin" is everyone's state, so `hide` stays 0 and everything is recorded.
- The server validates `to`: internal paths must be valid slug paths
("/" or `[a-z0-9_-]` segments), external ones are re-derived to the
https origin and accepted only when the client sent exactly that.
- The initial ping also records the visitor's `User-Agent` and
`Accept-Language` headers. The first `Accept-Language` tag is stored as
`lang` (e.g. `en-us`) and its region subtag, if present, is stored as
an initial `country` (e.g. `US`).
- The visitor IP is stored. A reverse-DNS lookup is attempted for each new
visit and the result, when available, is cached in RAM and stored as
`host`; local/reserved/multicast addresses are skipped.
- If a DB-IP MMDB file (`dbip-*.mmdb` or `dbip-*.mmdb.gz`) is present in the
repository root, it is loaded at startup and used to look up a more accurate
`country`. The MMDB lookup and the reverse-DNS lookup run in background
tasks after the visit is stored, so the `/ _a` response is never delayed.
The decompressed `dbip-*.mmdb` file is kept in the repository root and
ignored by git. The CLI flag `--dbip` (`uv run pagerite --dbip`) downloads
the latest `dbip-city-lite-YYYY-MM.mmdb.gz` from DB-IP before the server
starts, skipping the download when the local database is already current and
- **Client records**: the visitor's IP (IPv4 or IPv6 /64 network), raw
`User-Agent` and extracted `Accept-Language` tag are hashed with blake3;
the first 6 bytes identify a shared `Client` record. The `Client` stores
the full IP, `User-Agent`, compact `ua_pretty`, `lang`, initial
`country` from the language-region subtag, and asynchronously-filled
`country`/`city` from DB-IP geoip plus reverse-DNS `host`. Visits,
crawler hits and abuse hits all reference this record by its hash, so
client metadata is stored once instead of repeated per event.
- The visitor IP is stored in the `Client`. A reverse-DNS lookup is
attempted for each new client and the result, when available, is stored as
`host`; local/reserved/multicast addresses are skipped. If a DB-IP MMDB
file (`dbip-*.mmdb` or `dbip-*.mmdb.gz`) is present in the repository
root, it is loaded at startup and used to look up `country`/`city`. These
lookups run in background tasks after the event is stored, so the `/_a`
response is never delayed. The decompressed `dbip-*.mmdb` file is kept in
the repository root and ignored by git. The CLI flag `--dbip`
(`uv run pagerite --dbip`) downloads the latest
`dbip-city-lite-YYYY-MM.mmdb.gz` from DB-IP before the server starts,
skipping the download when the local database is already current and
removing older versions after an update; without the flag only an existing
file is used.
- **Crawler hits**: every document GET is queued in RAM as a pending crawler
hit. If a ping from the same (IP, User-Agent) pair arrives within 10
seconds the hit is discarded; otherwise it is written to `crawlers`.
Crawlers do not count as visits or views. In the analytics viewer, crawler
hits are grouped by the same (IP, User-Agent) pair and shown as a trail of
internal pages that crawler visited; the crawler table lists the most active
crawlers first rather than the most recent hits.
hit. If a ping from the same client arrives within 10 seconds the hit is
discarded; otherwise it is written to `crawlers`. Crawlers do not count as
visits or views. The `Accept-Language` header is stored on the shared
`Client` immediately; reverse-DNS host names and DB-IP geoip
country/city are filled in asynchronously, just like for real visits. In
the analytics viewer, crawler hits are grouped by client hash and shown as
a trail of internal pages that crawler visited; the crawler table lists
the most active crawlers first rather than the most recent hits.
- **Abuse (scanner) hits**: a 404 for a telltale path — any URL segment
starting with a dot (`/.env`, `/.git/config`) or ending in `.php` —
classifies the source IP as abuse immediately, and ten plain 404s from one
@@ -86,51 +92,55 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
IP is recorded as an abuse hit with the full request path (query string
included), and its pings are ignored. The classified IP set (`abuse_ips`)
is persisted in the JSON file; the plain-404 counters are RAM-only. In the
viewer, abuse hits are grouped by IP (never by UA — scanners randomize
theirs) in a separate "Abuse" table. Identical paths are collapsed into
one entry with their hit count; flagged paths that triggered classification
are lifted to the top, followed by other 404s and then document GETs from
the abuser. Raw User-Agent strings are shown one per line with their
occurrence counts, and the full lists are click-to-copy.
viewer, abuse hits are grouped by IP (never by client/UA — scanners
randomize theirs) in a separate "Abuse" table. Identical paths are
collapsed into one entry with their hit count; flagged paths that
triggered classification are lifted to the top, followed by other 404s and
then document GETs from the abuser. Raw User-Agent strings are shown one
per line with their occurrence counts, and the full lists are click-to-copy.
## Visits and sessions
There are no cookies. A visit is tied together by the (IP, User-Agent) pair
(IP from the first `X-Forwarded-For` hop — we sit behind a proxy — else the
direct peer): the first ping from a pair starts a new visit, subsequent
pings extend it. Pings arriving with no known session (server restart)
start a fresh visit from the first ping — treated as missing data rather
than dropped. The (IP, UA) → visit map and the IP → entry-referer/UTM
tables are in-memory only, but the IP and any resolvable reverse-DNS host
name are stored on the `Visit` record itself.
There are no cookies. A visit is tied together by a client hash — the first
6 bytes of a blake3 digest over the prettified IP (IPv4 unchanged, IPv6
/64 network), the raw `User-Agent` string and the extracted
`Accept-Language` tag. The first ping from a client hash starts a new
visit; subsequent pings extend it. Pings arriving with no known session
(server restart) start a fresh visit from the first ping — treated as
missing data rather than dropped. The client-hash → visit map and the IP →
entry-referer/UTM tables are in-memory only; client metadata is stored in
`Analytics.clients` keyed by the client hash.
Each `Client` record:
- `ip` — visitor IP address (first `X-Forwarded-For` hop, or direct peer),
- `host` — reverse-DNS host name for `ip` when resolvable, else `""`,
- `lang` — first `Accept-Language` tag, lowercased (e.g. `"en-us"`),
- `country` — two-letter country code. Initially derived from the
`Accept-Language` region subtag, but overwritten by the DB-IP MMDB result
when a database is available,
- `city` — city name from the DB-IP MMDB lookup, when available,
- `ua` — raw `User-Agent` string,
- `ua_pretty` — compact display form of the UA (browser/OS/device) when
parsable, otherwise the raw string.
Each `Visit` record:
- `start` — timestamp of the first event,
- `entry` — first page (path) seen,
- `referer` — external https origin of the initial load, `""` for direct,
- `ip` — visitor IP address (first `X-Forwarded-For` hop, or direct peer),
- `host` — reverse-DNS host name for `ip` when resolvable, else `""`,
- `client` — 6-byte blake3 hash referencing `Analytics.clients`,
- `trail` — everything seen afterwards in first-seen order: page paths and
external exit URLs. Re-visiting an already seen page (incl. the entry)
does not append.
- `lang` — first `Accept-Language` tag, lowercased (e.g. `en-us`),
- `country` — two-letter country code. Initially derived from the
`Accept-Language` region subtag, but overwritten by the DB-IP MMDB result
when a database is available,
- `city` — city name from the DB-IP MMDB lookup, when available,
- `ua` — raw `User-Agent` string from the initial ping,
- `ua_pretty` — compact display form of the UA (browser/OS/device) when
parsable, otherwise the raw string,
- `utm` — `utm_*` query parameters from the landing URL, as a dict.
- `read` — active reading time per path (seconds), keyed by path.
Each `CrawlerHit` record:
- `start` — timestamp of the document GET,
- `entry` — page path requested,
- `ip` — IP address,
- `ua` — raw `User-Agent` header,
- `ua_pretty` — compact display form of the UA when parsable,
- `client` — 6-byte blake3 hash referencing `Analytics.clients`,
- `referer` — external https origin of the request, `""` for direct/none,
- `query` — raw query string of the request.
@@ -138,20 +148,18 @@ Each `AbuseHit` record:
- `start` — timestamp of the request,
- `path` — full request path including the query string (e.g. `/.env?x=1`),
- `ip` — IP address (the grouping key for abusers),
- `ua` — raw `User-Agent` header,
- `ua_pretty` — compact display form of the UA when parsable,
- `client` — 6-byte blake3 hash referencing `Analytics.clients`,
- `flag` — true for the path that triggered abuse classification (telltale
path or the 404 that crossed the threshold),
- `is_404` — true for 404 responses, false for document GETs from the
abuser.
Crawler hits are grouped by (IP, User-Agent) in the analytics viewer; abuse
hits are grouped by IP alone. In the Abuse table identical paths are
collapsed with their counts; flagged paths that triggered classification are
lifted to the top, followed by other 404s and then document GETs from the
abuser. Within each category paths are sorted by count descending, then by
their earliest hit.
Crawler hits are grouped by client hash in the analytics viewer; abuse hits
are grouped by IP alone (resolved from the referenced `Client`). In the
Abuse table identical paths are collapsed with their counts; flagged paths
that triggered classification are lifted to the top, followed by other 404s
and then document GETs from the abuser. Within each category paths are
sorted by count descending, then by their earliest hit.
## Aggregates
frontend/src/AnalyticsView.vue
+31 -145
View File
@@ -16,8 +16,8 @@ import {
formatCrawlerRows,
formatVisitRows,
} from './analytics/format.js'
import * as flagSvgs from 'country-flag-icons/string/3x2'
import TrailLink from './TrailLink.vue'
import VisitorCell from './VisitorCell.vue'
import TransitionGraph from './TransitionGraph.vue'
import VisitorCharts from './VisitorCharts.vue'
@@ -91,23 +91,12 @@ watch(range, (r) => {
history.replaceState(null, '', url)
})
const visitRows = computed(() => formatVisitRows(visits.value, pageTree.value, now.value))
const clients = computed(() => data.value?.clients || {})
const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value))
const crawlers = computed(() => data.value?.crawlers || [])
const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, pageTree.value, now.value))
const abuseRows = computed(() => formatAbuseRows(data.value?.abuse || [], now.value))
const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value))
const abuseRows = computed(() => formatAbuseRows(data.value?.abuse || [], clients.value, now.value))
function flagSvg(code) {
return flagSvgs[code?.toUpperCase()] || ''
}
function countryName(code) {
if (!code) return ''
try {
return new Intl.DisplayNames(['en'], { type: 'region' }).of(code.toUpperCase())
} catch {
return ''
}
}
</script>
<template>
@@ -154,24 +143,17 @@ function countryName(code) {
<span v-if="v.utm && v.utm !== '—'" class="utm-tag small muted" :title="v.utmTitle">{{ v.utm }}</span>
<TrailLink v-for="(s, si) in v.trail" :key="si" :step="s" @close="$emit('close')" />
</td>
<td class="ip-locale-cell" :class="{ 'host-cell': v.isHost }">
<div class="ip-locale-rows">
<div class="ip-locale-row">
<div class="locale-line">
<span v-if="flagSvg(v.country)" class="flag" v-html="flagSvg(v.country)" :title="countryName(v.country) || v.country"></span>
<template v-if="v.city && v.city !== '—'"><small class="city-name muted">{{ v.city }}</small></template>
<template v-else-if="!flagSvg(v.country)">—</template>
</div>
<div class="ip-line"><span class="clickable-ip small muted"
:title="v.ip"
@click="copyIp(v.ip, $event)">{{ v.ipDisplay }}</span></div>
</div>
<div class="ip-locale-row">
<div class="ua-line"><small class="muted" :title="v.uaRaw">{{ v.ua }}</small></div>
<div v-if="v.lang && v.lang !== '—'" class="locale-lang"><small class="muted">{{ v.langDisplay }}</small></div>
</div>
</div>
</td>
<VisitorCell
:ip="v.ip"
:ip-display="v.ipDisplay"
:ua="v.ua"
:ua-raw="v.uaRaw"
:country="v.country"
:city="v.city"
:lang="v.lang"
:lang-display="v.langDisplay"
:is-host="v.isHost"
/>
<td class="last-seen muted"
:title="v.lastSeenLocal"
@click="copyList(v.lastSeenIso, $event)">{{ v.lastSeen }}</td>
@@ -189,7 +171,7 @@ function countryName(code) {
<thead>
<tr>
<th>pages</th>
<th>ip / ua</th>
<th>visitor</th>
<th class="last-seen">last seen</th>
</tr>
</thead>
@@ -198,12 +180,17 @@ function countryName(code) {
<td class="trail">
<TrailLink v-for="(s, si) in c.pages" :key="si" :step="s" :count="s.count" @close="$emit('close')" />
</td>
<td class="ip-ua-cell">
<div><span class="clickable-ip small muted"
:title="c.ip"
@click="copyIp(c.ip, $event)">{{ c.ipDisplay }}</span></div>
<div class="ua-line"><small class="muted" :title="c.uaRaw">{{ c.ua }}</small></div>
</td>
<VisitorCell
:ip="c.ip"
:ip-display="c.ipDisplay"
:ua="c.ua"
:ua-raw="c.uaRaw"
:country="c.country"
:city="c.city"
:lang="c.lang"
:lang-display="c.langDisplay"
:is-host="c.isHost"
/>
<td class="last-seen muted"
:title="c.lastSeenLocal"
@click="copyList(c.lastSeenIso, $event)">{{ c.lastSeen }}</td>
@@ -221,7 +208,7 @@ function countryName(code) {
<thead>
<tr>
<th>paths</th>
<th>ip / uas</th>
<th>visitor</th>
<th class="last-seen">last seen</th>
</tr>
</thead>
@@ -440,20 +427,13 @@ function countryName(code) {
hyphens: none;
}
.visit-table .clickable-ip,
.visit-table :deep(.clickable-ip),
.visit-table .clickable-list,
.visit-table .last-seen {
cursor: pointer;
position: relative;
}
.visit-table .ip-locale-cell {
width: 36ch;
max-width: 36ch;
overflow: hidden;
text-overflow: ellipsis;
}
.visit-table .ip-ua-cell {
width: 22ch;
max-width: 22ch;
@@ -462,79 +442,7 @@ function countryName(code) {
text-overflow: ellipsis;
}
.visit-table .host-cell {
text-align: right;
}
.visit-table .ip-locale-rows {
display: flex;
flex-direction: column;
gap: 0.15rem;
}
.visit-table .ip-locale-row {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.5rem;
}
.visit-table .ip-locale-row > * {
min-width: 0;
}
.visit-table .ip-locale-row .locale-line,
.visit-table .ip-locale-row .ip-line,
.visit-table .ip-locale-row .ua-line {
flex: 1 1 auto;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.visit-table .ip-locale-row .locale-lang {
flex: 0 1 auto;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
text-align: right;
}
.visit-table .ip-locale-row .locale-line {
text-align: left;
}
.visit-table .ip-locale-row .ip-line {
text-align: right;
}
.visit-table .ip-locale-row .ua-line {
text-align: left;
}
.visit-table .locale-line {
display: flex;
align-items: center;
gap: 0.3rem;
}
.visit-table .city-name {
display: inline-block;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
vertical-align: middle;
}
.visit-table .ua-line {
text-align: right;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.visit-table .copy-popup {
.visit-table :deep(.copy-popup) {
position: absolute;
bottom: calc(100% + 0.25rem);
left: 50%;
@@ -549,28 +457,6 @@ function countryName(code) {
z-index: 10;
}
.visit-table .locale-line .flag {
display: inline-flex;
width: 18px;
height: 12px;
border-radius: 2px;
overflow: hidden;
border: 1px solid var(--line);
box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
vertical-align: middle;
}
.visit-table .locale-line .flag :deep(svg) {
width: 100%;
height: 100%;
display: block;
}
.visit-table .locale-line .city-name {
margin-left: 0.3rem;
vertical-align: middle;
}
.crawler-top-uas {
font-size: 0.9rem;
margin-bottom: 0.6rem;
frontend/src/TrailLink.vue
+28
View File
@@ -0,0 +1,28 @@
<script setup>
import { formatCount } from './analytics/format.js'
defineProps({
step: { type: Object, required: true },
count: { type: Number, default: 0 },
})
defineEmits(['close'])
</script>
<template>
<a class="trail-link"
:href="step.path"
:title="count > 1 ? `${step.title} (${count} hits)` : step.title"
:target="step.external ? '_blank' : undefined"
:rel="step.external ? 'noopener' : undefined"
@click="(e) => { if (!step.external) $emit('close') }">
<small v-if="count > 1" class="muted">{{ formatCount(count) }}×</small>
<span :class="{ desat: step.home }">{{ step.slug }}</span>
</a>
</template>
<style scoped>
.desat {
filter: saturate(0);
}
</style>
frontend/src/VisitorCell.vue
+150
View File
@@ -0,0 +1,150 @@
<script setup>
// Visitor metadata cell shared by the recent-visits and crawlers tables.
// Displays IP/network/host, country flag/city, UA, and language when available.
// Clicking the IP copies the full address to the clipboard.
import { computed } from 'vue'
import * as flagSvgs from 'country-flag-icons/string/3x2'
import { copyIp, formatLang } from './analytics/format.js'
const props = defineProps({
ip: { type: String, default: '' },
ipDisplay: { type: String, default: '—' },
ua: { type: String, default: '' },
uaRaw: { type: String, default: '' },
country: { type: String, default: '' },
city: { type: String, default: '' },
lang: { type: String, default: '' },
langDisplay: { type: String, default: '' },
isHost: { type: Boolean, default: false },
})
const hasCountry = computed(() => !!(props.country && props.country !== '—'))
const hasCity = computed(() => !!(props.city && props.city !== '—'))
const hasLocale = computed(() => hasCountry.value || hasCity.value)
const langValue = computed(() => props.langDisplay || formatLang(props.lang))
const showLang = computed(() => langValue.value && langValue.value !== '—')
function flagSvg(code) {
return flagSvgs[code?.toUpperCase()] || ''
}
function countryName(code) {
if (!code) return ''
try {
return new Intl.DisplayNames(['en'], { type: 'region' }).of(code.toUpperCase())
} catch {
return ''
}
}
</script>
<template>
<td class="visitor-cell" :class="{ 'host-cell': isHost }">
<div class="visitor-rows">
<div class="visitor-row">
<div class="locale-line">
<span v-if="flagSvg(country)" class="flag" v-html="flagSvg(country)" :title="countryName(country) || country"></span>
<template v-if="hasCity"><small class="city-name muted">{{ city }}</small></template>
<template v-else-if="!hasLocale">—</template>
</div>
<div class="ip-line">
<span class="clickable-ip small muted"
:title="ip"
@click="copyIp(ip, $event)">{{ ipDisplay }}</span>
</div>
</div>
<div class="visitor-row">
<div class="ua-line"><small class="muted" :title="uaRaw">{{ ua || '—' }}</small></div>
<div v-if="showLang" class="locale-lang"><small class="muted">{{ langValue }}</small></div>
</div>
</div>
</td>
</template>
<style scoped>
.visitor-cell {
width: 18em;
max-width: 18em;
overflow: hidden;
text-overflow: ellipsis;
vertical-align: top;
}
.visitor-cell.host-cell {
text-align: right;
}
.visitor-rows {
display: flex;
flex-direction: column;
gap: 0.15rem;
}
.visitor-row {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.5rem;
}
.visitor-row > * {
min-width: 0;
}
.locale-line,
.ip-line,
.ua-line {
flex: 1 1 auto;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.locale-line {
text-align: left;
display: flex;
align-items: center;
gap: 0.3rem;
}
.ip-line {
text-align: right;
}
.ua-line {
text-align: left;
}
.locale-lang {
flex: 0 0 auto;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
text-align: right;
}
.city-name {
display: inline-block;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
vertical-align: middle;
}
.flag {
display: inline-flex;
width: 18px;
height: 12px;
border-radius: 2px;
overflow: hidden;
border: 1px solid var(--line);
box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
vertical-align: middle;
}
.flag :deep(svg) {
width: 100%;
height: 100%;
display: block;
}
</style>
frontend/src/analytics/format.js
+77 -61
View File
@@ -305,32 +305,52 @@ export function formatCounts(entries) {
/**
* Count distinct User-Agent strings among crawler hits, most common first.
* Returns an array of [ua, count] pairs.
* Returns an array of [ua, count] pairs. ``clients`` maps client hashes to
* client records.
*/
export function countCrawlerUas(crawlers) {
export function countCrawlerUas(crawlers, clients) {
const counts = {}
for (const c of crawlers || []) {
const value = c.ua_pretty || c.ua || '(no UA)'
const client = (clients || {})[c.client] || {}
const value = client.ua_pretty || client.ua || '(no UA)'
counts[value] = (counts[value] || 0) + 1
}
return Object.entries(counts).sort((a, b) => b[1] - a[1])
}
/**
* Group raw crawler hits by the same (ip, ua) pair we use to tell a real
* visitor from a crawler, and format each group as a row showing every
* internal page that crawler visited. Rows are sorted by total hits,
* most active crawler first, rather than by most recent hit.
* Reduce a reverse-DNS hostname to its right-most components that fit
* within ``limit`` characters. This keeps the meaningful main domain
* while avoiding absurdly long subdomains like ``xxx.yyy.zzz...provider.net``.
*/
export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
export function mainDomain(host, limit = 24) {
if (!host) return host
const labels = host.split('.').filter(Boolean)
if (!labels.length) return host
const parts = [labels.pop()]
while (labels.length) {
const next = labels[labels.length - 1]
const candidate = `${next}.${parts.join('.')}`
if (candidate.length > limit) break
parts.unshift(labels.pop())
}
return parts.join('.')
}
/**
* Group raw crawler hits by client hash and format each group as a row showing
* every internal page that crawler visited. Rows are sorted by total hits,
* most active crawler first, rather than by most recent hit.
* ``clients`` maps client hashes to client records.
*/
export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) {
const titles = buildTitleMap(pageTree)
const groups = new Map()
for (const c of crawlers || []) {
const key = `${c.ip}\0${c.ua}`
const g = groups.get(key) || {
ip: c.ip || '',
ua: c.ua_pretty || c.ua || '—',
uaRaw: c.ua || '',
const client = (clients || {})[c.client] || {}
const g = groups.get(c.client) || {
clientHash: c.client,
client,
lastStart: 0,
pages: new Map(),
}
@@ -339,7 +359,7 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
if (c.entry?.startsWith('/')) {
g.pages.set(c.entry, (g.pages.get(c.entry) || 0) + 1)
}
groups.set(key, g)
groups.set(c.client, g)
}
const totalHits = (g) => {
let n = 0
@@ -349,19 +369,29 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
return [...groups.values()]
.sort((a, b) => totalHits(b) - totalHits(a) || b.lastStart - a.lastStart)
.slice(0, 10)
.map((g) => ({
lastSeen: formatWhen(g.lastStart, now),
lastSeenIso: formatWhenIso(g.lastStart),
lastSeenLocal: formatWhenLocal(g.lastStart),
pages: [...g.pages.entries()]
.sort((a, b) => b[1] - a[1])
.map(([path, count]) => ({ ...stepOf(path, titles), count })),
ip: g.ip,
ipDisplay: hostIP(g.ip) || g.ip || '—',
ua: g.ua,
uaRaw: g.uaRaw,
total: totalHits(g),
}))
.map((g) => {
const client = g.client || {}
const host = client.host || ''
const isHost = !!host
return {
lastSeen: formatWhen(g.lastStart, now),
lastSeenIso: formatWhenIso(g.lastStart),
lastSeenLocal: formatWhenLocal(g.lastStart),
pages: [...g.pages.entries()]
.sort((a, b) => b[1] - a[1])
.map(([path, count]) => ({ ...stepOf(path, titles), count })),
ip: client.ip || '',
ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip) || client.ip || '—',
isHost,
ua: client.ua_pretty || client.ua || '—',
uaRaw: client.ua || '',
lang: client.lang || '—',
langDisplay: formatLang(client.lang),
country: client.country || '—',
city: client.city || '—',
total: totalHits(g),
}
})
}
/**
@@ -374,12 +404,15 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
* count descending, then earliest first. UAs are shown raw, one per line,
* with their occurrence counts. Paths are shown verbatim (query string
* included), not resolved against the page tree.
* ``clients`` maps client hashes to client records.
*/
export function formatAbuseRows(abuse, now = Date.now()) {
export function formatAbuseRows(abuse, clients, now = Date.now()) {
const groups = new Map()
for (const a of abuse || []) {
const g = groups.get(a.ip) || {
ip: a.ip || '',
const client = (clients || {})[a.client] || {}
const ip = client.ip || ''
const g = groups.get(ip) || {
ip,
pathCounts: new Map(),
rawUas: [],
uaCounts: new Map(),
@@ -400,10 +433,10 @@ export function formatAbuseRows(abuse, now = Date.now()) {
if (a.flag) existing.flag = true
if (!a.is_404) existing.is_404 = false
g.pathCounts.set(path, existing)
const ua = a.ua || '(no UA)'
const ua = client.ua || '(no UA)'
g.rawUas.push(ua)
g.uaCounts.set(ua, (g.uaCounts.get(ua) || 0) + 1)
groups.set(a.ip, g)
groups.set(ip, g)
}
const totalHits = (g) => {
let n = 0
@@ -447,34 +480,17 @@ export function formatAbuseRows(abuse, now = Date.now()) {
})
}
/**
* Reduce a reverse-DNS hostname to its right-most components that fit
* within ``limit`` characters. This keeps the meaningful main domain
* while avoiding absurdly long subdomains like ``xxx.yyy.zzz...provider.net``.
*/
function mainDomain(host, limit = 24) {
if (!host) return host
const labels = host.split('.').filter(Boolean)
if (!labels.length) return host
const parts = [labels.pop()]
while (labels.length) {
const next = labels[labels.length - 1]
const candidate = `${next}.${parts.join('.')}`
if (candidate.length > limit) break
parts.unshift(labels.pop())
}
return parts.join('.')
}
/**
* Format raw visit records as rows for a technical table. Returns objects
* with display strings; missing values become "—". ``trail`` starts with the
* external referer (when present), then the entry page and any further internal
* pages or external exit origins. Only the 20 most recent visits are shown.
* ``clients`` maps client hashes to client records.
*/
export function formatVisitRows(visits, pageTree, now = Date.now()) {
export function formatVisitRows(visits, clients, pageTree, now = Date.now()) {
const titles = buildTitleMap(pageTree)
return [...(visits || [])].reverse().slice(0, 20).map((v) => {
const client = (clients || {})[v.client] || {}
const trail = [v.entry, ...(v.trail || [])]
.map((p) => stepOf(p, titles))
.filter(Boolean)
@@ -485,24 +501,24 @@ export function formatVisitRows(visits, pageTree, now = Date.now()) {
.map(([k, value]) => `${k}=${value}`)
.join(', ')
const dash = (s) => (s || '—')
const host = v.host || ''
const host = client.host || ''
const isHost = !!host
return {
lastSeen: formatWhen(v.start, now),
lastSeenIso: formatWhenIso(v.start),
lastSeenLocal: formatWhenLocal(v.start),
langDisplay: formatLang(v.lang),
langDisplay: formatLang(client.lang),
trail,
refererStep: stepOf(v.referer, titles),
referer: dash(v.referer),
ip: v.ip || '',
ipDisplay: isHost ? mainDomain(host) : hostIP(v.ip) || v.ip || '—',
ip: client.ip || '',
ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip) || client.ip || '—',
isHost,
lang: dash(v.lang),
country: dash(v.country),
city: dash(v.city),
ua: v.ua_pretty || v.ua || '—',
uaRaw: v.ua || '',
lang: dash(client.lang),
country: dash(client.country),
city: dash(client.city),
ua: client.ua_pretty || client.ua || '—',
uaRaw: client.ua || '',
utm: utm || '—',
utmTitle,
}
pagerite/analytics.py
+212 -130
View File
@@ -10,14 +10,16 @@ Admin clients ping with ``hide=1``, which records nothing and removes any
visit the session accumulated before logging in. Scanner telltale 404s
(dotpaths, *.php) classify the source IP as abuse; its hits — including
earlier crawler hits — are moved to the abuse list, which the viewer
groups by IP with full request paths. The session map is in-memory only.
The visitor IP and, when available, its reverse-DNS host name are stored
on the visit record itself.
groups by IP with full request paths. Client metadata (IP, UA, language,
country/city, host) is stored once per unique client hash and referenced
from visits, crawler hits and abuse hits. The session map is in-memory
only.
Data is a msgspec Struct JSON-dumped to its own file (not the kanta db),
rewritten atomically on every recorded event.
"""
import ipaddress
import os
import re
import tempfile
@@ -27,6 +29,7 @@ from datetime import UTC, datetime, timedelta
from pathlib import Path
from urllib.parse import parse_qs, urlparse
import blake3
import msgspec
from ua_parser import parse
@@ -56,34 +59,46 @@ def _compact_user_agent(ua: str) -> str:
return " ".join(p for p in parts if p).strip()
class Client(msgspec.Struct, omit_defaults=True):
"""Client metadata shared by visits, crawler hits and abuse hits.
Identified by a 6-byte blake3 hash of the IPv4 address or IPv6 /64
network, the full User-Agent string and the extracted language tag.
Country/city/host are filled in asynchronously after the first event.
"""
#: Visitor IP address (first X-Forwarded-For hop or direct peer).
ip: str = ""
#: Reverse-DNS host name for ``ip`` when resolvable, else "".
host: str = ""
#: First Accept-Language tag, lowercased (e.g. "en-us").
lang: str = ""
#: Two-letter country code from the DB-IP geoip lookup, or "".
country: str = ""
#: City name from the DB-IP geoip lookup, or "".
city: str = ""
#: Raw User-Agent header.
ua: str = ""
#: Compact display form of ``ua`` (browser/OS/device) when parsable.
ua_pretty: str = ""
class Visit(msgspec.Struct, omit_defaults=True):
"""One visit: the initial-load data plus everything seen afterwards.
``trail`` holds page paths and external exit URLs in first-seen
order; re-visiting an already seen page does not append. The entry
page itself is in ``entry``, not in the trail.
page itself is in ``entry``, not in the trail. Client metadata is
held in ``Analytics.clients`` keyed by ``client``.
"""
start: datetime
entry: str
#: External https origin of the initial load, "" for direct visits.
referer: str = ""
#: Visitor IP address (first X-Forwarded-For hop or direct peer).
ip: str = ""
#: Reverse-DNS host name for ``ip`` when resolvable, else "".
host: str = ""
#: 6-byte blake3 hash referencing ``Analytics.clients``.
client: bytes = b""
trail: list[str] = []
#: First Accept-Language tag, lowercased (e.g. "en-us").
lang: str = ""
#: Two-letter region subtag derived from ``lang`` (e.g. "US"), or "".
#: Overwritten by the DB-IP geoip lookup when a database is available.
country: str = ""
#: City name from the DB-IP geoip lookup, or "".
city: str = ""
#: Raw User-Agent header from the initial ping.
ua: str = ""
#: Compact display form of ``ua`` (browser/OS/device) when parsable.
ua_pretty: str = ""
#: UTM query parameters from the landing URL, keyed by parameter name.
utm: dict[str, str] = {}
#: Active reading time per path (seconds), keyed by path.
@@ -91,14 +106,15 @@ class Visit(msgspec.Struct, omit_defaults=True):
class CrawlerHit(msgspec.Struct, omit_defaults=True):
"""A document GET that was never followed by an analytics ping."""
"""A document GET that was never followed by an analytics ping.
Client metadata is held in ``Analytics.clients`` keyed by ``client``.
"""
start: datetime
entry: str
ip: str = ""
ua: str = ""
#: Compact display form of ``ua`` when parsable.
ua_pretty: str = ""
#: 6-byte blake3 hash referencing ``Analytics.clients``.
client: bytes = b""
#: External https origin of the initial load, "" for direct/none.
referer: str = ""
#: Raw query string of the landing URL (UTM tags can be parsed from it).
@@ -112,15 +128,14 @@ class AbuseHit(msgspec.Struct, omit_defaults=True):
kept: the interesting part is exactly which paths were probed.
``flag`` marks the path that triggered classification; ``is_404``
distinguishes 404 responses from document GETs made by the abuser.
Client metadata is held in ``Analytics.clients`` keyed by ``client``.
"""
start: datetime
#: Full request path including the query string (e.g. "/.env?x=1").
path: str
ip: str = ""
ua: str = ""
#: Compact display form of ``ua`` when parsable.
ua_pretty: str = ""
#: 6-byte blake3 hash referencing ``Analytics.clients``.
client: bytes = b""
#: True when this path triggered abuse classification (telltale path
#: or the 404 that crossed the threshold).
flag: bool = False
@@ -137,6 +152,8 @@ class Analytics(msgspec.Struct, omit_defaults=True):
crawlers: list[CrawlerHit] = []
#: Requests from abusive IPs (see AbuseHit), grouped by IP in the viewer.
abuse: list[AbuseHit] = []
#: Client metadata keyed by 6-byte blake3 hash.
clients: dict[bytes, Client] = {}
#: IPs classified as scanners/abusers (keys; values always True).
abuse_ips: dict[str, bool] = {}
#: Page transitions per 5-minute bucket (sparse):
@@ -236,8 +253,36 @@ def _is_abuse_path(path: str) -> bool:
return bool(_ABUSE_PATH.search(path.split("?")[0]))
def _network_ip(ip: str) -> str:
"""IPv4 address unchanged, IPv6 collapsed to its /64 network address.
We hash the network rather than the full address so that clients in the
same /64 (a typical end-user allocation) are treated as one visitor.
"""
if not ip:
return ip
try:
addr = ipaddress.ip_address(ip)
except ValueError:
return ip
if isinstance(addr, ipaddress.IPv6Address):
return str(ipaddress.IPv6Network(f"{ip}/64", strict=False).network_address)
return ip
def _client_hash(ip: str, ua: str, lang: str) -> bytes:
"""6-byte blake3 digest identifying a visitor/client tuple.
The key is the prettified IP (IPv6 /64), the raw UA string and the
extracted language tag, separated by null bytes.
"""
return blake3.blake3(
f"{_network_ip(ip)}\0{ua}\0{lang}".encode()
).digest()[:6]
class Store:
"""In-memory analytics data plus the (IP, UA) -> visit session map."""
"""In-memory analytics data plus the client-hash -> visit session map."""
def __init__(self, path: Path) -> None:
self.path = path
@@ -247,8 +292,8 @@ class Store:
self.data = msgspec.json.decode(path.read_bytes(), type=Analytics)
except msgspec.DecodeError, OSError:
pass # legacy schema / corrupt or unreadable file: start fresh
#: (ip, user-agent) -> index of the current visit in data.visits
self.sessions: dict[tuple[str, str], int] = {}
#: client hash -> index of the current visit in data.visits
self.sessions: dict[bytes, int] = {}
#: ip -> external https origin of the latest document GET carrying
#: one, stashed for the visit the client's initial ping starts.
#: Internal or absent referers never touch the table.
@@ -296,20 +341,26 @@ class Store:
else:
self._notify()
def _flush_crawlers(self, now: datetime | None = None) -> None:
"""Move expired pending crawler hits into persistent ``data.crawlers``."""
def _flush_crawlers(self, now: datetime | None = None) -> list[bytes]:
"""Move expired pending crawler hits into persistent ``data.crawlers``.
Returns the client hashes of the newly flushed hits so callers can
schedule async enrichment.
"""
if not self.pending_crawlers:
return
return []
now = now or datetime.now(UTC)
cutoff = now - _CRAWLER_TIMEOUT
expired: list[CrawlerHit] = []
remaining: list[CrawlerHit] = []
for hit in self.pending_crawlers:
(expired if hit.start <= cutoff else remaining).append(hit)
if expired:
self.pending_crawlers = remaining
self.data.crawlers.extend(expired)
self._save()
if not expired:
return []
self.pending_crawlers = remaining
self.data.crawlers.extend(expired)
self._save()
return [hit.client for hit in expired]
def _count(self, table: dict[str, int], key: str) -> None:
table[key] = table.get(key, 0) + 1
@@ -357,24 +408,72 @@ class Store:
if i > index:
self.sessions[key] = i - 1
def _abuse_hit(
def _client_ip(self, client_hash: bytes) -> str:
"""Return the IP stored for ``client_hash``, or "" if missing."""
client = self.data.clients.get(client_hash)
return client.ip if client else ""
def _ensure_client(
self,
ip: str,
ua: str,
lang: str,
*,
country: str = "",
) -> bytes:
"""Get or create a ``Client`` record; return its 6-byte hash."""
h = _client_hash(ip, ua, lang)
if h not in self.data.clients:
self.data.clients[h] = Client(
ip=ip,
ua=ua,
ua_pretty=_compact_user_agent(ua),
lang=lang,
country=country,
)
self._save()
return h
def enrich_client(
self,
client_hash: bytes,
*,
host: str = "",
country: str = "",
city: str = "",
) -> None:
"""Fill in host/geoip fields on a client record after async lookups."""
client = self.data.clients.get(client_hash)
if client is None:
return
changed = False
if host and not client.host:
client.host = host
changed = True
if country:
client.country = country
changed = True
if city:
client.city = city
changed = True
if changed:
self._save()
def _abuse_hit(
self,
client_hash: bytes,
path: str,
start: datetime | None = None,
*,
flag: bool = False,
is_404: bool = False,
) -> None:
"""Append one abuse hit with the full request path."""
"""Append one abuse hit referencing a client by hash."""
self.data.abuse.append(
AbuseHit(
start=start or datetime.now(UTC),
path=path,
ip=ip,
ua=ua,
ua_pretty=_compact_user_agent(ua),
client=client_hash,
flag=flag,
is_404=is_404,
)
@@ -383,7 +482,7 @@ class Store:
def classify_abuse(
self,
ip: str,
ua: str,
client_hash: bytes,
path: str,
*,
flag: bool = False,
@@ -397,54 +496,62 @@ class Store:
"""
if ip not in self.data.abuse_ips:
self.data.abuse_ips[ip] = True
moved = [h for h in self.data.crawlers if h.ip == ip]
moved = [h for h in self.data.crawlers if self._client_ip(h.client) == ip]
if moved:
self.data.crawlers = [h for h in self.data.crawlers if h.ip != ip]
self.data.crawlers = [h for h in self.data.crawlers if self._client_ip(h.client) != ip]
for h in moved:
self._abuse_hit(
h.ip, h.ua,
h.client,
h.entry + (f"?{h.query}" if h.query else ""),
start=h.start,
)
pending = [h for h in self.pending_crawlers if h.ip == ip]
pending = [h for h in self.pending_crawlers if self._client_ip(h.client) == ip]
if pending:
self.pending_crawlers = [h for h in self.pending_crawlers if h.ip != ip]
self.pending_crawlers = [h for h in self.pending_crawlers if self._client_ip(h.client) != ip]
for h in pending:
self._abuse_hit(
h.ip, h.ua,
h.client,
h.entry + (f"?{h.query}" if h.query else ""),
start=h.start,
)
self._abuse_hit(ip, ua, path, flag=flag, is_404=is_404)
self._abuse_hit(client_hash, path, flag=flag, is_404=is_404)
self._save()
def track_404(self, ip: str, ua: str, path: str) -> None:
def track_404(
self,
ip: str,
ua: str,
path: str,
accept_language: str = "",
) -> bytes:
"""Record a 404 response for ``path`` (full path, query included).
A telltale path (dot segment or *.php) classifies the IP as abuse
immediately; enough plain 404s from one IP do too. Hits from
already-classified IPs go straight to the abuse list.
Returns the client hash so callers can schedule async enrichment.
"""
lang, country = _parse_accept_language(accept_language)
client_hash = self._ensure_client(ip, ua, lang, country=country)
if ip in self.data.abuse_ips:
self._abuse_hit(ip, ua, path, flag=_is_abuse_path(path), is_404=True)
self._abuse_hit(client_hash, path, flag=_is_abuse_path(path), is_404=True)
self._save()
return
return client_hash
if _is_abuse_path(path):
self.classify_abuse(ip, ua, path, flag=True, is_404=True)
return
self.classify_abuse(ip, client_hash, path, flag=True, is_404=True)
return client_hash
self.not_found_counts[ip] = self.not_found_counts.get(ip, 0) + 1
if self.not_found_counts[ip] >= _ABUSE_404_THRESHOLD:
self.classify_abuse(ip, ua, path, flag=True, is_404=True)
self.classify_abuse(ip, client_hash, path, flag=True, is_404=True)
return client_hash
return client_hash
def _new_visit(
self,
entry: str,
referer: str,
key: tuple[str, str],
ip: str = "",
lang: str = "",
country: str = "",
ua: str = "",
client_hash: bytes,
utm: dict[str, str] | None = None,
) -> Visit:
now = datetime.now(UTC)
@@ -452,45 +559,16 @@ class Store:
start=now,
entry=entry,
referer=referer,
ip=ip,
lang=lang,
country=country,
ua=ua,
ua_pretty=_compact_user_agent(ua),
client=client_hash,
utm=utm or {},
)
self.data.visits.append(visit)
self.sessions[key] = len(self.data.visits) - 1
self.sessions[client_hash] = len(self.data.visits) - 1
self._count(self.data.site_visits, _bucket(now))
self._count(self.data.views.setdefault(entry, {}), _bucket(now))
self._count_transition(referer or "(direct)", entry, now)
return visit
def enrich_visit(
self,
index: int,
*,
host: str = "",
country: str = "",
city: str = "",
) -> None:
"""Fill in host/geoip fields on an existing visit after async lookups."""
if index < 0 or index >= len(self.data.visits):
return
visit = self.data.visits[index]
changed = False
if host and not visit.host:
visit.host = host
changed = True
if country:
visit.country = country
changed = True
if city:
visit.city = city
changed = True
if changed:
self._save()
def track_entry(
self,
referer: str,
@@ -498,7 +576,8 @@ class Store:
ip: str,
ua: str,
full_path: str,
) -> None:
accept_language: str = "",
) -> list[bytes]:
"""Stash the entry referer/UTM tags and queue a pending crawler hit.
Nothing is counted here — the client's initial /_a ping starts the
@@ -509,21 +588,28 @@ class Store:
does not erase an earlier tagged landing.
Every document GET is also queued as a pending crawler hit. If a ping
from the same (IP, UA) pair arrives within ``_CRAWLER_TIMEOUT``, the
hit is discarded; otherwise it is flushed to ``data.crawlers``.
from the same client arrives within ``_CRAWLER_TIMEOUT``, the hit is
discarded; otherwise it is flushed to ``data.crawlers``. The
Accept-Language header is stored on the client record immediately;
host/geoip are filled in later by async enrichment.
GETs from IPs already classified as abuse are recorded as abuse hits
with the full request path (query string included).
Returns the client hashes of any hits flushed to persistent storage,
so callers can schedule async enrichment.
"""
entry = full_path.split("?")[0]
query = full_path.split("?", 1)[1] if "?" in full_path else ""
lang, country = _parse_accept_language(accept_language)
client_hash = self._ensure_client(ip, ua, lang, country=country)
if ip in self.data.abuse_ips:
self._flush_crawlers()
self._abuse_hit(ip, ua, full_path, is_404=False, flag=False)
flushed = self._flush_crawlers()
self._abuse_hit(client_hash, full_path, is_404=False, flag=False)
self._save()
return
return flushed
now = datetime.now(UTC)
self._flush_crawlers(now)
flushed = self._flush_crawlers(now)
if referer:
origin = _origin(referer)
if origin is not None and origin != own_origin:
@@ -535,19 +621,18 @@ class Store:
CrawlerHit(
start=now,
entry=entry,
ip=ip,
ua=ua,
ua_pretty=_compact_user_agent(ua),
client=client_hash,
referer=self.pending_referers.get(ip, ""),
query=query,
)
)
return flushed
def _add_read(self, ip: str, ua: str, path: str, seconds: int) -> None:
def _add_read(self, client_hash: bytes, path: str, seconds: int) -> None:
"""Add ``seconds`` of reading time for ``path`` to the current visit."""
if seconds <= 0:
return
index = self.sessions.get((ip, ua))
index = self.sessions.get(client_hash)
if index is None or index >= len(self.data.visits):
return
visit = self.data.visits[index]
@@ -562,7 +647,7 @@ class Store:
accept_language: str = "",
hide: bool = False,
read: int = 0,
) -> int | None:
) -> tuple[int | None, list[bytes]]:
"""Record a client navigation ping ({from, to, read} from pagerite.js).
``to`` is an internal path ("/...") or an https URL for exit links; a
@@ -575,63 +660,59 @@ class Store:
referer and UTM tags stashed by the document GET if there are any.
``hide`` is set by admin clients: the ping cancels pending crawler
hits as usual, and any existing visit for this (IP, UA) session is
hits as usual, and any existing visit for this client session is
removed from the stats (the admin browsed anonymously before logging
in). Nothing new is recorded.
Pings from IPs classified as abuse are ignored entirely.
Returns the index of the new visit when one is created, so callers
can enrich it later with non-blocking lookups (host, geoip country).
Returns the index of the new visit when one is created (or None) and
the client hashes of any crawler hits flushed by this call, so callers
can schedule async enrichment (host, geoip country/city).
"""
self._flush_crawlers()
key = (ip, ua)
flushed = self._flush_crawlers()
lang, country = _parse_accept_language(accept_language)
client_hash = _client_hash(ip, ua, lang)
if hide:
# Admin ping: cancel pending crawler hits and scrub the session.
self.pending_crawlers = [
hit for hit in self.pending_crawlers if not (hit.ip == ip and hit.ua == ua)
hit for hit in self.pending_crawlers if hit.client == client_hash
]
index = self.sessions.pop(key, None)
index = self.sessions.pop(client_hash, None)
if index is not None and index < len(self.data.visits):
self._remove_visit(index)
self._save()
return None
return None, flushed
if ip in self.data.abuse_ips:
return None
# A real visitor ping cancels any pending crawler hits from this
# (IP, UA) pair.
return None, flushed
# A real visitor ping cancels any pending crawler hits from this client.
self.pending_crawlers = [
hit for hit in self.pending_crawlers if not (hit.ip == ip and hit.ua == ua)
hit for hit in self.pending_crawlers if hit.client != client_hash
]
fr_path = _internal_path(from_) if from_ else ""
if fr_path and read > 0:
self._add_read(ip, ua, fr_path, read)
self._add_read(client_hash, fr_path, read)
if not to:
if read > 0:
self._save()
return None
return None, flushed
if to.startswith("/") and not to.startswith("//"):
target = _internal_path(to) or ""
else:
target = _external_target(to) or ""
if not target:
return None
key = (ip, ua)
index = self.sessions.get(key)
return None, flushed
index = self.sessions.get(client_hash)
fr = fr_path or "(direct)"
if index is None or index >= len(self.data.visits):
# No known session: the initial ping of a fresh page load (or
# missing data after a server restart) — start a visit.
lang, country = _parse_accept_language(accept_language)
index = len(self.data.visits)
self._ensure_client(ip, ua, lang, country=country)
self._new_visit(
target,
self.pending_referers.pop(ip, ""),
key,
ip=ip,
lang=lang,
country=country,
ua=ua,
client_hash,
utm=self.pending_utms.pop(ip, {}),
)
else:
@@ -644,4 +725,5 @@ class Store:
if visit.entry != target and target not in visit.trail:
visit.trail.append(target)
self._save()
return index if index is not None and index < len(self.data.visits) else None
visit_index = index if index is not None and index < len(self.data.visits) else None
return visit_index, flushed
pagerite/app.py
+42 -20
View File
@@ -652,14 +652,21 @@ async def _geoip_city(ip: str) -> str:
return await asyncio.to_thread(_geoip.city, ip)
async def _enrich_visit(index: int, ip: str) -> None:
"""Run non-blocking reverse-DNS and geoip enrichment for a new visit."""
if not ip:
async def _enrich_client(client_hash: bytes) -> None:
"""Run non-blocking reverse-DNS and geoip enrichment for a client."""
client = analytics_store.data.clients.get(client_hash)
if not client or not client.ip:
return
host = await _lookup_host(ip)
country = await _geoip_country(ip)
city = await _geoip_city(ip)
analytics_store.enrich_visit(index, host=host, country=country, city=city)
host = await _lookup_host(client.ip)
country = await _geoip_country(client.ip)
city = await _geoip_city(client.ip)
analytics_store.enrich_client(client_hash, host=host, country=country, city=city)
def _schedule_client_enrichment(client_hashes: list[bytes]) -> None:
"""Start background host/geoip enrichment for the given client hashes."""
for client_hash in client_hashes:
asyncio.create_task(_enrich_client(client_hash))
async def _broadcast_analytics() -> None:
@@ -728,7 +735,7 @@ async def analytics_ping(ping: AnalyticsPing, request: Request) -> None:
the response is never delayed by slow DNS or the first MMDB decompress.
"""
ip = _client_ip(request)
index = analytics_store.ping(
visit_index, flushed_clients = analytics_store.ping(
ping.fr,
ping.to,
ip,
@@ -737,11 +744,13 @@ async def analytics_ping(ping: AnalyticsPing, request: Request) -> None:
hide=bool(ping.hide),
read=ping.read,
)
if index is not None:
asyncio.create_task(_enrich_visit(index, ip))
if visit_index is not None:
visit = analytics_store.data.visits[visit_index]
asyncio.create_task(_enrich_client(visit.client))
_schedule_client_enrichment(flushed_clients)
def _track_entry(path: str, request: Request) -> None:
def _track_entry(path: str, request: Request) -> list[bytes]:
"""Stash the referer/UTM tags and queue a pending crawler hit for the GET.
Nothing is counted on the GET itself — the client's /_a ping starts the
@@ -752,21 +761,25 @@ def _track_entry(path: str, request: Request) -> None:
``127.0.0.1``) is ignored: it is not real traffic and would otherwise be
logged as a crawler hit. The root-path and localhost checks prevent
remote visitors from hiding traffic with the same query string.
Returns the client hashes of any pending crawler hits flushed to persistent
storage, so callers can schedule async geoip and reverse-DNS enrichment.
"""
if (
path == ""
and str(request.url.query) == "from=devserver.py"
and _client_ip(request) == "127.0.0.1"
):
return
return []
own_origin = f"https://{urlparse(str(request.base_url)).netloc}"
full_path = f"{request.url.path}{_query_suffix(request)}"
analytics_store.track_entry(
return analytics_store.track_entry(
request.headers.get("referer", ""),
own_origin,
_client_ip(request),
request.headers.get("user-agent", ""),
full_path,
request.headers.get("accept-language", ""),
)
@@ -986,16 +999,20 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
placeholder page (nav links point straight at its first child).
"""
path = path.strip("/")
ua = request.headers.get("user-agent", "")
accept_language = request.headers.get("accept-language", "")
if path and _is_reserved(path):
# Invalid slug shape: not a content URL, let FastAPI return its
# built-in 404 instead of rendering an editable article page.
# Scanner telltales (dotpaths like /.env, *.php) classify the IP
# as abuse in analytics.
analytics_store.track_404(
client_hash = analytics_store.track_404(
_client_ip(request),
request.headers.get("user-agent", ""),
ua,
f"/{path}{_query_suffix(request)}",
accept_language,
)
asyncio.create_task(_enrich_client(client_hash))
raise HTTPException(404)
chain = resolve(data.menu, path)
node = chain[-1] if chain else None
@@ -1010,7 +1027,8 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
if request.headers.get("if-none-match") == etag:
return Response(status_code=304)
if _is_trackable_path(path):
_track_entry(path, request)
flushed = _track_entry(path, request)
_schedule_client_enrichment(flushed)
return HTMLResponse(
views.render_page(data.menu, path, data.brand, data.custom_css, data.theme, data.favicon, data.brand_html, str(request.base_url).rstrip("/")),
headers={
@@ -1023,7 +1041,8 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
# Category label without a landing page: placeholder with the pen
# to create it (404 — no page here, but the node is real).
if _is_trackable_path(path):
_track_entry(path, request)
flushed = _track_entry(path, request)
_schedule_client_enrichment(flushed)
return HTMLResponse(
views.render_category(data.menu, path, data.brand, data.custom_css, data.theme, data.favicon, data.brand_html),
404,
@@ -1039,10 +1058,13 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
if item.published:
return RedirectResponse(f"/{slug}")
if _is_trackable_path(path):
analytics_store.track_404(
client_hash = analytics_store.track_404(
_client_ip(request),
request.headers.get("user-agent", ""),
ua,
f"/{path}{_query_suffix(request)}",
accept_language,
)
_track_entry(path, request)
asyncio.create_task(_enrich_client(client_hash))
flushed = _track_entry(path, request)
_schedule_client_enrichment(flushed)
return HTMLResponse(views.render_not_found(data.menu, path, data.brand, data.custom_css, data.theme, data.favicon, data.brand_html), 404)