Add --dbip CLI flag to auto-download/update the DB-IP MMDB database.

Downloads the latest dbip-city-lite-YYYY-MM.mmdb.gz before starting the server, skipping when the local database is current, falling back to the previous month on 404, and removing older databases after an update. Promotes httpx to a runtime dependency.
Default scheme, host and port for fake_traffic script.
2026-08-21 03:09:04 +00:00 · 2026-08-21 02:52:53 +00:00 · 2026-08-21 02:50:36 +00:00 · 2026-08-21 02:49:18 +00:00 · 2026-08-21 02:44:54 +00:00 · 2026-08-21 01:53:39 +00:00
16 changed files with 638 additions and 179 deletions
@@ -8,8 +8,8 @@ Struct dumped to disk — separate from the kanta content database, path from
 - `pagerite/analytics.py` — data model (`Analytics`, `Visit`) and the `Store`
  (in-memory data + session map, atomic JSON persistence).
 - `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`),
-  the `POST /_a` ping endpoint, and `GET /_api/analytics` (admin-gated like
-  every `/_api` endpoint).
+  the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics`
+  (admin-gated like every `/_api` endpoint).
 - `frontend/src/pagerite.js` — client navigation pings and the 📊 pen.
 - `frontend/src/AnalyticsView.vue` — viewer component rendered inside the
  normal site layout on the `/_a` analytics page.
@@ -32,9 +32,11 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
 - **Internal fetch-navigations**: `to` is the target path, sent only after
  the swap actually happened (a failed swap falls back to a full load,
  whose initial ping counts the view instead — no gap, no double count).
- **External links** (`https` only): `to` is the link's origin. This is the
+- **External links** (`https` only): `to` is the link's full URL. This is the
  exit-link record; the user may continue navigating afterwards (new tab,
-  back), so the exit origin is not necessarily the last trail entry.
+  back), so the exit URL is not necessarily the last trail entry. Outbound
+  links are stored by full URL so several links to the same domain remain
+  distinct.
 - **Excluded**: back/forward (popstate) navigations, navigation involving
  the analytics page itself (`/_a`), and everything while the user is known to
  be an admin *and SSO is actually in use* — with no auth proxy (dev/test)
@@ -55,11 +57,18 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
  `country`.  The MMDB lookup and the reverse-DNS lookup run in background
  tasks after the visit is stored, so the `/ _a` response is never delayed.
  The decompressed `dbip-*.mmdb` file is kept in the repository root and
-  ignored by git.
+  ignored by git.  The CLI flag `--dbip` (`uv run pagerite --dbip`) downloads
+  the latest `dbip-city-lite-YYYY-MM.mmdb.gz` from DB-IP before the server
+  starts, skipping the download when the local database is already current and
+  removing older versions after an update; without the flag only an existing
+  file is used.
 - **Crawler hits**: every document GET is queued in RAM as a pending crawler
  hit.  If a ping from the same (IP, User-Agent) pair arrives within 10
  seconds the hit is discarded; otherwise it is written to `crawlers`.
-  Crawlers do not count as visits or views.
+  Crawlers do not count as visits or views.  In the analytics viewer, crawler
+  hits are grouped by the same (IP, User-Agent) pair and shown as a trail of
+  internal pages that crawler visited; the crawler table lists the most active
+  crawlers first rather than the most recent hits.

 ## Visits and sessions

@@ -80,12 +89,13 @@ Each `Visit` record:
 - `ip` — visitor IP address (first `X-Forwarded-For` hop, or direct peer),
 - `host` — reverse-DNS host name for `ip` when resolvable, else `""`,
 - `trail` — everything seen afterwards in first-seen order: page paths and
-  external exit origins. Re-visiting an already seen page (incl. the entry)
+  external exit URLs. Re-visiting an already seen page (incl. the entry)
  does not append.
 - `lang` — first `Accept-Language` tag, lowercased (e.g. `en-us`),
 - `country` — two-letter country code.  Initially derived from the
  `Accept-Language` region subtag, but overwritten by the DB-IP MMDB result
  when a database is available,
+- `city` — city name from the DB-IP MMDB lookup, when available,
 - `ua` — raw `User-Agent` string from the initial ping,
 - `ua_pretty` — compact display form of the UA (browser/OS/device) when
  parsable, otherwise the raw string,
@@ -131,9 +141,9 @@ The 📊 pen in the banner corner (admins only, injected by pagerite.js next to
 the edit pens) links to `/_a`, the analytics page. It is a normal site page:
 the standard banner, navigation and footer stay in place, and the analytics
 content is rendered inside `#main`. The page itself is public, but the data
-still comes from `GET /_api/analytics`, which remains admin-gated like the
-rest of the management API; visitors without access see the viewer with a
-"could not be loaded" message.
+stream comes from `WebSocket /_api/ws/analytics`, which remains admin-gated
+like the rest of the management API; visitors without access see the viewer
+with a "could not be loaded" message.

 Because it is a real page, fetch-navigation handles it like any other internal
 link: clicking the 📊 pen (or any link to `/_a`) fetches the server-rendered
@@ -156,16 +166,19 @@ the smoothing time scale follows the unit: the month+ sigmas are 24× the
 hourly ones. The y max is derived from the smoothed curves so single-bucket
 spikes don't blow up the scale, and raw spikes are clamped into the plot.
 Axes always start at 0 and end at a multiple of a 1-2-5 major step (max 5
-labeled intervals, minor lines at fifths when integral; the floor is 1/h).
+labeled intervals, minor lines at fifths when integral; the minimum y-axis
+range is 10 so tiny values such as a single visit are not stretched to a
+fractional scale).
 The week range is aligned to Monday 00:00 UTC and overlays up to 8 previous
 weeks in the same accent color at decreasing opacity (the current week is
 truncated at the current bucket, never drawing fake zeroes for the future);
 its x labels are weekday names centered at midday UTC, without vertical grid
 lines (day boundaries would be misleading in the viewer's timezone). The
 month view labels days the same lineless way — day numbers at noon UTC,
-with the month name substituted for the 1st. Year and all are rolling
-windows ending at now, re-bucketed to daily points, with boundary lines at
-months/years. Below the charts: a radial **transition map** (all pages from
+with the month name substituted for the 1st. Year is a rolling 365-day window ending at now, re-bucketed to daily points,
+with boundary lines at months/years.  All uses the full data reach, but keeps
+at least the past 30 days so the chart never collapses to a tiny sliver when
+the site is young. Below the charts: a radial **transition map** (all pages from
 `/_api/pages` — front page at the center, each slug level on its own ring,
 siblings clockwise in navigation order from the top, radial gap equal to
 the arc spacing — opposite transition directions joined into organic
@@ -178,5 +191,7 @@ to the directional count with no in-flight limit, opposing directions
 offset onto parallel lanes. External referers show as a node row above the
 map, external exits as small nodes fanned outwards from their source
 page), per-page view
-counts, the top transitions and the 50 most recent visit trails. Data comes from `GET /_api/analytics`, which
-returns the raw JSON file contents.
+counts, the top transitions and the 50 most recent visit trails. Data is
+streamed live over `WebSocket /_api/ws/analytics`, which pushes the latest
+JSON snapshot on connect and again whenever the analytics file is updated
+(with a small server-side debounce to avoid flooding under high traffic).
@@ -1,16 +1,14 @@
 <script setup>
-// Analytics viewer rendered as a normal page inside #main. Fetches the raw
-// collected data from /_api/analytics (admin-gated by the auth proxy) and
+// Analytics viewer rendered as a normal page inside #main. Receives live
+// analytics data over /_api/ws/analytics (admin-gated by the auth proxy) and
 // renders totals, smoothed visit/views curves, a transition map, and recent
 // visit/crawler tables. Read-only.
 // See docs/analytics.md for the data format.
-import { computed, onMounted, ref, watch } from 'vue'
+import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
 import { RANGES } from './analytics/time.js'
 import {
  calcTotalViews,
  copyIp,
-  countCrawlerUas,
-  formatCounts,
  formatCrawlerRows,
  formatVisitRows,
 } from './analytics/format.js'
@@ -25,23 +23,54 @@ const props = defineProps({
 const data = ref(null)
 const pageTree = ref(null)
 const error = ref('')
+const now = ref(Date.now())
+let ws = null
+let reconnectTimeout = null
+let timeInterval = null

-onMounted(async () => {
-  try {
-    const res = await fetch('/_api/analytics')
-    if (!res.ok) throw new Error(res.statusText)
-    data.value = await res.json()
-  } catch {
+function connectAnalytics() {
+  if (ws) return
+  const proto = location.protocol === 'https:' ? 'wss:' : 'ws:'
+  ws = new WebSocket(`${proto}//${location.host}/_api/ws/analytics`)
+  ws.onopen = () => { error.value = '' }
+  ws.onmessage = (event) => {
+    try {
+      data.value = JSON.parse(event.data)
+    } catch {
+      error.value = 'analytics data could not be loaded'
+    }
+  }
+  ws.onerror = () => {
    error.value = 'analytics data could not be loaded'
  }
+  ws.onclose = () => {
+    ws = null
+    reconnectTimeout = setTimeout(connectAnalytics, 2000)
+  }
+}
+
+onMounted(async () => {
+  connectAnalytics()
+  now.value = Date.now()
+  timeInterval = setInterval(() => { now.value = Date.now() }, 30000)
  // The site tree for the transition map (all pages in menu order). Not
-  // fatal: without it the map falls back to transition endpoints only.
+  // fatal: without it the map just narrows to pages seen in transitions.
  try {
    const res = await fetch('/_api/pages')
    if (res.ok) pageTree.value = await res.json()
  } catch { /* map just narrows to pages seen in transitions */ }
 })

+onUnmounted(() => {
+  if (reconnectTimeout) clearTimeout(reconnectTimeout)
+  if (timeInterval) clearInterval(timeInterval)
+  if (ws) {
+    ws.onclose = null
+    ws.close()
+    ws = null
+  }
+})
+
 const visits = computed(() => data.value?.visits || [])
 const totalViews = computed(() => calcTotalViews(data.value?.views))

@@ -54,10 +83,9 @@ watch(range, (r) => {
  history.replaceState(null, '', url)
 })

-const visitRows = computed(() => formatVisitRows(visits.value, pageTree.value))
+const visitRows = computed(() => formatVisitRows(visits.value, pageTree.value, now.value))
 const crawlers = computed(() => data.value?.crawlers || [])
-const crawlerRows = computed(() => formatCrawlerRows(crawlers.value))
-const topCrawlerUas = computed(() => countCrawlerUas(crawlers.value).slice(0, 10))
+const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, pageTree.value, now.value))

 function flagSvg(code) {
  return flagSvgs[code?.toUpperCase()] || ''
@@ -115,10 +143,13 @@ function countryName(code) {
              </thead>
              <tbody>
                <tr v-for="(v, i) in visitRows" :key="i">
-                  <td class="when">{{ v.when }}</td>
+                  <td class="when" :title="v.whenTooltip">{{ v.when }}</td>
                  <td class="trail">
                    <a v-for="(s, si) in v.trail" :key="si"
-                       :href="s.path" :title="s.title" @click="emit('close')">
+                       :href="s.path" :title="s.title"
+                       :target="s.external ? '_blank' : undefined"
+                       :rel="s.external ? 'noopener' : undefined"
+                       @click="(e) => { if (!s.external) $emit('close') }">
                      {{ s.slug }}
                    </a>
                  </td>
@@ -131,6 +162,8 @@ function countryName(code) {
                  <td>{{ v.lang }}</td>
                  <td class="country">
                    <span v-if="flagSvg(v.country)" class="flag" v-html="flagSvg(v.country)" :title="countryName(v.country) || v.country"></span>
+                    <template v-if="v.city !== '—'">{{ countryName(v.country) || v.country }}<br><small class="muted">{{ v.city }}</small></template>
+                    <template v-else-if="v.country !== '—'">{{ countryName(v.country) || v.country }}</template>
                    <template v-else>—</template>
                  </td>
                  <td class="ua" :title="v.uaRaw">{{ v.ua }}</td>
@@ -144,33 +177,32 @@ function countryName(code) {

        <section>
          <h2>Crawlers</h2>
-          <div v-if="topCrawlerUas.length" class="crawler-top-uas">
-            <p><strong>top UAs:</strong> {{ formatCounts(topCrawlerUas) }}</p>
-          </div>
          <div v-if="crawlerRows.length" class="visit-table-wrap">
            <table class="visit-table">
              <thead>
                <tr>
                  <th>when</th>
-                  <th>entry</th>
+                  <th>pages</th>
                  <th>ip</th>
                  <th>ua</th>
-                  <th>referer</th>
-                  <th>query</th>
                </tr>
              </thead>
              <tbody>
                <tr v-for="(c, i) in crawlerRows" :key="i">
-                  <td class="when">{{ c.when }}</td>
-                  <td>{{ c.entry }}</td>
+                  <td class="when" :title="c.whenTooltip">{{ c.when }}</td>
+                  <td class="trail">
+                    <a v-for="(s, si) in c.pages" :key="si"
+                       :href="s.path" :title="`${s.title}${s.count > 1 ? ` (${s.count} hits)` : ''}`"
+                       @click="$emit('close')">
+                      <small v-if="s.count > 1" class="muted">{{ s.count }}×</small>{{ s.slug }}
+                    </a>
+                  </td>
                  <td>
                    <span class="clickable-ip"
                          :title="`Click to copy full IP: ${c.ip}`"
                          @click="copyIp(c.ip)">{{ c.ipDisplay }}</span>
                  </td>
                  <td class="ua" :title="c.uaRaw">{{ c.ua }}</td>
-                  <td>{{ c.referer }}</td>
-                  <td>{{ c.query }}</td>
                </tr>
              </tbody>
            </table>
@@ -307,6 +339,12 @@ function countryName(code) {
  margin-left: 0.5rem;
 }

+.visit-table .trail small,
+.visit-table small.muted {
+  color: var(--muted);
+  font-size: 0.75em;
+}
+
 .visit-table .clickable-ip {
  cursor: pointer;
  text-decoration: underline;
@@ -109,10 +109,12 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
      <circle v-for="(b, i) in beads" :key="'b' + i"
              :cx="b.x" :cy="b.y" :r="BEAD_R" class="tbead" />
      <g v-for="(x, i) in graph.extNodes" :key="'x' + i">
-        <circle :cx="x.x" :cy="x.y" :r="x.r" class="txnode">
-          <title>{{ x.path }}</title>
-        </circle>
-        <text :x="x.x" :y="x.y + x.r + 11" class="txlabel">{{ x.label }}</text>
+        <a :href="x.path" target="_blank" rel="noopener" :title="x.path">
+          <circle :cx="x.x" :cy="x.y" :r="x.r"
+                  :class="['txnode', x.kind === 'source' ? 'txnode-source' : 'txnode-exit']" />
+          <text :x="x.x" :y="x.y - 2" class="tnodeslug">{{ x.label }}</text>
+          <text :x="x.x" :y="x.y + 12" class="tnodecount">{{ x.count }}</text>
+        </a>
      </g>
      <g v-for="n in graph.nodes" :key="n.path">
        <a :href="n.path" :title="n.title">
@@ -144,14 +146,10 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
 }
 .tmap .txnode {
  fill: var(--bg, Canvas);
-  stroke: var(--muted);
-  stroke-width: 1;
-}
-.tmap .txlabel {
-  fill: var(--muted);
-  font-size: 9px;
-  text-anchor: middle;
+  stroke-width: 1.5;
 }
+.tmap .txnode-source { stroke: var(--text); }
+.tmap .txnode-exit { stroke: var(--muted); }
 .tmap .tarc {
  fill: none;
  stroke: var(--line);
@@ -14,9 +14,9 @@ export const PAD_TOP = 14 // room above the highest point
 /**
 * Y always starts at 0; the max is a multiple of a 1-2-5 major step with at
 * most 5 intervals, so labeled ticks are always round and evenly divided.
- * Values are per-unit rates, so small scales are legitimate (a lone visit
- * smoothes to well under 1/unit) — the floor is 1, not 10. Minor lines
- * subdivide each major step in five when that yields integers.
+ * A minimum range of 10 keeps tiny near-zero values (e.g. a single visit)
+ * from being enlarged to a fractional scale; minor lines subdivide each
+ * major step in five when that yields integers.
 */
 export function yScale(maxValue) {
  let step = 1
@@ -27,9 +27,9 @@ export function yScale(maxValue) {
    }
  }
  let max = Math.ceil(maxValue / step) * step
-  if (max < 1) {
-    max = 1
-    step = 0.5
+  if (max < 10) {
+    max = 10
+    step = 2
  }
  const minor = step >= 5 && step % 5 === 0 ? step / 5 : null
  return { max, step, minor }
@@ -62,10 +62,89 @@ function slugOf(path) {
  return path === '/' ? '🏠' : path.split('/').pop()
 }

+/** Host name of an external https origin, with scheme stripped. */
+function externalSlug(origin) {
+  try {
+    return new URL(origin).host
+  } catch {
+    return origin.replace(/^https?:\/\//, '')
+  }
+}
+
+/** Format one trail step: an internal page or an external https origin. */
+function stepOf(path, titles) {
+  if (path?.startsWith('/')) {
+    return { path, slug: slugOf(path), title: titles.get(path) || '', external: false }
+  }
+  if (path?.startsWith('https://')) {
+    return {
+      path,
+      slug: externalSlug(path),
+      title: 'External site',
+      external: true,
+    }
+  }
+  return null
+}
+
+/**
+ * Human-readable relative timestamp.  Adapted from cista-storage: uses
+ * ``Intl.RelativeTimeFormat`` for short intervals and a compact date for
+ * anything older than a week.
+ */
+export function formatWhen(ts, now = Date.now()) {
+  const date = new Date(ts)
+  const diff = date.getTime() - now
+  const adiff = Math.abs(diff)
+  const formatter = new Intl.RelativeTimeFormat('en', { numeric: 'auto' })
+  if (adiff <= 5000) return 'now'
+  if (adiff <= 60000) {
+    return formatter
+      .format(Math.round(diff / 1000), 'second')
+      .replace(' ago', '')
+      .replaceAll(' ', '\u202F')
+  }
+  if (adiff <= 3600000) {
+    return formatter
+      .format(Math.round(diff / 60000), 'minute')
+      .replace('utes', '')
+      .replace('ute', '')
+      .replaceAll(' ', '\u202F')
+  }
+  if (adiff <= 86400000) {
+    return formatter
+      .format(Math.round(diff / 3600000), 'hour')
+      .replaceAll(' ', '\u202F')
+  }
+  if (adiff <= 604800000) {
+    return formatter
+      .format(Math.round(diff / 86400000), 'day')
+      .replaceAll(' ', '\u202F')
+  }
+  let d = date
+    .toLocaleDateString('en-ie', {
+      weekday: 'short',
+      year: 'numeric',
+      month: 'short',
+      day: 'numeric',
+    })
+    .replace('Sept', 'Sep')
+  if (d.length === 14) d = d.replace(' ', ' \u2007')
+  d = d.replaceAll(' ', '\u202F').replace('\u202F', '\u00A0')
+  d = d.slice(0, -4) + d.slice(-2)
+  return d
+}
+
+/** Full UTC timestamp for tooltips, e.g. "2026-08-21 00:20:48 UTC". */
+export function formatWhenTooltip(ts) {
+  return new Date(ts).toISOString().replace('T', ' ').replace('Z', ' UTC')
+}
+
 /**
 * Format recent visits for display, newest first. Each step is a linked slug
- * pointing to its article; external referers/origins and direct entries are
- * omitted. The link title shows the article heading when known.
+ * pointing to its article; external referers/origins are shown as their
+ * domain name with the full origin as the link href. The link title shows the
+ * article heading when known, or "External site" for origins.
 */
 export function formatRecentVisits(visits, pageTree, limit = 50) {
  const titles = buildTitleMap(pageTree)
@@ -73,13 +152,9 @@ export function formatRecentVisits(visits, pageTree, limit = 50) {
    .reverse()
    .map((v) => ({
      when: new Date(v.start).toLocaleString(),
-      steps: [v.entry, ...(v.trail || [])]
-        .filter((p) => p?.startsWith('/'))
-        .map((p) => ({
-          path: p,
-          slug: slugOf(p),
-          title: titles.get(p) || '',
-        })),
+      steps: [v.referer, v.entry, ...(v.trail || [])]
+        .map((p) => stepOf(p, titles))
+        .filter(Boolean),
    }))
    .filter((v) => v.steps.length)
    .slice(0, limit)
@@ -133,44 +208,76 @@ export function countCrawlerUas(crawlers) {
 }

 /**
- * Format raw crawler hit records as rows for a technical table.  Missing
- * values become "—".
+ * Group raw crawler hits by the same (ip, ua) pair we use to tell a real
+ * visitor from a crawler, and format each group as a row showing every
+ * internal page that crawler visited.  Rows are sorted by total hits,
+ * most active crawler first, rather than by most recent hit.
 */
-export function formatCrawlerRows(crawlers) {
-  const dash = (s) => (s || '—')
-  return [...(crawlers || [])].reverse().map((c) => ({
-    when: new Date(c.start).toLocaleString(),
-    entry: dash(c.entry),
-    ip: c.ip || '',
-    ipDisplay: c.host || hostIP(c.ip) || c.ip || '—',
-    ua: c.ua_pretty || c.ua || '—',
-    uaRaw: c.ua || '',
-    referer: dash(c.referer),
-    query: dash(c.query),
-  }))
+export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
+  const titles = buildTitleMap(pageTree)
+  const groups = new Map()
+  for (const c of crawlers || []) {
+    const key = `${c.ip}\0${c.ua}`
+    const g = groups.get(key) || {
+      ip: c.ip || '',
+      ua: c.ua_pretty || c.ua || '—',
+      uaRaw: c.ua || '',
+      lastStart: 0,
+      pages: new Map(),
+    }
+    const start = new Date(c.start).getTime()
+    if (start > g.lastStart) g.lastStart = start
+    if (c.entry?.startsWith('/')) {
+      g.pages.set(c.entry, (g.pages.get(c.entry) || 0) + 1)
+    }
+    groups.set(key, g)
+  }
+  const totalHits = (g) => {
+    let n = 0
+    for (const c of g.pages.values()) n += c
+    return n
+  }
+  return [...groups.values()]
+    .sort((a, b) => totalHits(b) - totalHits(a) || b.lastStart - a.lastStart)
+    .slice(0, 10)
+    .map((g) => ({
+      when: formatWhen(g.lastStart, now),
+      whenTooltip: formatWhenTooltip(g.lastStart),
+      pages: [...g.pages.entries()]
+        .sort((a, b) => b[1] - a[1])
+        .map(([path, count]) => ({
+          path,
+          slug: slugOf(path),
+          title: titles.get(path) || '',
+          count,
+        })),
+      ip: g.ip,
+      ipDisplay: hostIP(g.ip) || g.ip || '—',
+      ua: g.ua,
+      uaRaw: g.uaRaw,
+      total: totalHits(g),
+    }))
 }

 /**
 * Format raw visit records as rows for a technical table.  Returns objects
- * with display strings; missing values become "—".  ``trail`` joins page
- * titles (when known) with " -> ".
+ * with display strings; missing values become "—".  ``trail`` starts with the
+ * external referer (when present), then the entry page and any further internal
+ * pages or external exit origins. Only the 20 most recent visits are shown.
 */
-export function formatVisitRows(visits, pageTree) {
+export function formatVisitRows(visits, pageTree, now = Date.now()) {
  const titles = buildTitleMap(pageTree)
-  return [...(visits || [])].reverse().map((v) => {
-    const trail = [v.entry, ...(v.trail || [])]
-      .filter((p) => p?.startsWith('/'))
-      .map((p) => ({
-        path: p,
-        slug: slugOf(p),
-        title: titles.get(p) || '',
-      }))
+  return [...(visits || [])].reverse().slice(0, 20).map((v) => {
+    const trail = [v.referer, v.entry, ...(v.trail || [])]
+      .map((p) => stepOf(p, titles))
+      .filter(Boolean)
    const utm = Object.entries(v.utm || {})
      .map(([k, value]) => `${k}=${value}`)
      .join(', ')
    const dash = (s) => (s || '—')
    return {
-      when: new Date(v.start).toLocaleString(),
+      when: formatWhen(v.start, now),
+      whenTooltip: formatWhenTooltip(v.start),
      trail,
      referer: dash(v.referer),
      ip: v.ip || '',
@@ -178,6 +285,7 @@ export function formatVisitRows(visits, pageTree) {
      host: dash(v.host),
      lang: dash(v.lang),
      country: dash(v.country),
+      city: dash(v.city),
      ua: v.ua_pretty || v.ua || '—',
      uaRaw: v.ua || '',
      utm: utm || '—',
@@ -16,7 +16,7 @@ export const RANGES = {
  week: { label: 'week' },
  month: { label: 'month', span: 30 * DAY, bucket: 6 * HOUR },
  year: { label: 'year', span: 365 * DAY, bucket: DAY },
-  all: { label: 'all', span: null, bucket: DAY },
+  all: { label: 'all', span: null, bucket: DAY, minSpan: 30 * DAY },
 }

 /** Monday 00:00 UTC of the week containing t (epoch day 0 was a Thursday). */
@@ -89,16 +89,19 @@ export function weeklySeries(buckets) {
 /**
 * Rolling window for the non-week ranges (x max = now), counts converted
 * to per-day rates (the unit the month+ charts are read in).
+ * Ranges without a fixed span use the full data reach, but never less than
+ * their configured minSpan so the chart keeps a readable minimum x scale.
 */
 export function rollingSeries(buckets, rangeKey) {
  const raw = rawTimes(buckets)
  const times = Object.keys(raw).map(Number)
  if (!times.length) return null
-  const { span, bucket } = RANGES[rangeKey]
+  const { span, bucket, minSpan = 0 } = RANGES[rangeKey]
  const t1 = Math.floor(Date.now() / bucket) * bucket + bucket
+  const earliest = Math.floor(Math.min(...times) / bucket) * bucket
  const t0 = span != null
    ? t1 - span
-    : Math.floor(Math.min(...times) / bucket) * bucket
+    : Math.min(earliest, t1 - minSpan)
  const points = []
  for (let t = t0; t < t1; t += bucket) {
    points.push({ t, count: sumRange(raw, t, t + bucket) })
@@ -14,12 +14,13 @@
 * emitted at time intervals inversely proportional (linear) to the
 * directional count.
 * External referers appear as nodes in a row above the map, external exits
- * as small nodes just outside their source page, angled away from the
- * center. Self-loops (reload pings) are skipped.
+ * as full-size nodes just outside their source page, angled away from the
+ * center. Each distinct full exit URL is its own node. Self-loops (reload
+ * pings) are skipped.
 */

 export const TNODE_R = 34 // node circles hold the slug and the view count
-export const EXT_R = 16 // external referer/exit nodes
+export const EXT_R = 34 // external referer/exit nodes use the same full size

 // Edge width (half-width of the thin middle) grows logarithmically with
 // the count, anchored so a single recorded transition renders as a ~1 px
@@ -86,7 +87,7 @@ function collectInternalTransitions(transitions) {
 /** Short display label for an external origin (protocol stripped). */
 function extLabel(ext) {
  const s = ext.replace(/^https?:\/\//, '')
-  return s.length > 18 ? `${s.slice(0, 17)}…` : s
+  return s.length > 11 ? `${s.slice(0, 10)}…` : s
 }

 /**
@@ -231,11 +232,33 @@ function buildFamilyArcs(nodes, radius) {
    arcs.push({
      d: `M ${Math.cos(a0) * r} ${Math.sin(a0) * r} `
       + `A ${r} ${r} 0 ${large} 1 ${Math.cos(a1) * r} ${Math.sin(a1) * r}`,
+      r,
+      a0,
+      a1,
    })
  }
  return arcs
 }

+/** Bounding box of a circular arc centred at the origin, sampled. */
+function arcBounds(r, a0, a1) {
+  let x0 = Infinity
+  let y0 = Infinity
+  let x1 = -Infinity
+  let y1 = -Infinity
+  const steps = 36
+  for (let i = 0; i <= steps; i++) {
+    const t = a0 + (a1 - a0) * (i / steps)
+    const x = Math.cos(t) * r
+    const y = Math.sin(t) * r
+    if (x < x0) x0 = x
+    if (y < y0) y0 = y
+    if (x > x1) x1 = x
+    if (y > y1) y1 = y
+  }
+  return { x0, y0, x1, y1 }
+}
+
 /** Collapse opposite transition directions into one unordered pair per page pair. */
 function aggregatePairs(internal) {
  const pairs = new Map() // unordered pair key -> [countAB, countBA]
@@ -519,7 +542,8 @@ function buildExternal(external, byPath, radius, innerBounds) {
    const spacing = 2 * EXT_R + 44
    const x0 = cx - ((origins.length - 1) * spacing) / 2
    origins.forEach(({ ext, ps }, i) => {
-      const xn = { path: ext, label: extLabel(ext), x: x0 + i * spacing, y, r: EXT_R }
+      const total = ps.reduce((s, p) => s + p.in, 0)
+      const xn = { path: ext, label: extLabel(ext), x: x0 + i * spacing, y, r: EXT_R, count: total, kind: 'source' }
      extNodes.push(xn)
      for (const p of ps) {
        const page = byPath.get(p.page)
@@ -529,30 +553,46 @@ function buildExternal(external, byPath, radius, innerBounds) {
    })
  }

-  // Outgoing: small exit nodes fanned outwards from the source page.
+  // Outgoing: group by full URL so several links to the same domain stay
+  // distinct. Each exit node is placed one ring-gap outside its source page
+  // (same radial spacing internal rings use), fanned around the source angle,
+  // and shows the total count across all pages that link to that URL.
+  const GAP = radius(1) - radius(0)
  const outgoing = live.filter((p) => p.out >= minCount)
-    .sort((a, b) => b.out - a.out).slice(0, MAX_EXT_OUT)
+    .sort((a, b) => b.out - a.out)
  const perPage = new Map()
+  const selected = []
  for (const p of outgoing) {
    const used = perPage.get(p.page) || 0
    if (used >= MAX_EXT_OUT_PER_PAGE) continue
    perPage.set(p.page, used + 1)
+    selected.push(p)
+    if (selected.length >= MAX_EXT_OUT) break
+  }
+
+  const exitNodes = new Map() // full URL -> node
+  const placedPerPage = new Map() // for angle fanning of the placement anchor
+  for (const p of selected) {
    const page = byPath.get(p.page)
-    // Fan multiple exits of one page symmetrically around the outward
-    // direction; the center page has no angle, so its exits point down
-    // (the top row above the map belongs to referers).
-    const base = page.depth ? page.angle : Math.PI / 2
-    const ang = base + [0, 0.4, -0.4][used]
-    let dist = TNODE_R + 40
-    let x = page.x + Math.cos(ang) * dist
-    let y = page.y + Math.sin(ang) * dist
-    for (let tries = 0; tries < 5 && overlaps(x, y, EXT_R); tries++) {
-      dist += 24
-      x = page.x + Math.cos(ang) * dist
-      y = page.y + Math.sin(ang) * dist
+    let xn = exitNodes.get(p.ext)
+    if (!xn) {
+      const used = placedPerPage.get(p.page) || 0
+      placedPerPage.set(p.page, used + 1)
+      const base = page.depth ? page.angle : Math.PI / 2
+      const ang = base + [0, 0.4, -0.4][used]
+      let dist = GAP
+      let x = page.x + Math.cos(ang) * dist
+      let y = page.y + Math.sin(ang) * dist
+      for (let tries = 0; tries < 5 && overlaps(x, y, EXT_R); tries++) {
+        dist += GAP * 0.3
+        x = page.x + Math.cos(ang) * dist
+        y = page.y + Math.sin(ang) * dist
+      }
+      xn = { path: p.ext, label: extLabel(p.ext), x, y, r: EXT_R, count: 0, kind: 'exit' }
+      exitNodes.set(p.ext, xn)
+      extNodes.push(xn)
    }
-    const xn = { path: p.ext, label: extLabel(p.ext), x, y, r: EXT_R }
-    extNodes.push(xn)
+    xn.count += p.out
    edges.push(buildRibbon(page, xn, p.out, 0, width(p.out), TNODE_R, EXT_R))
    flows.push(...buildFlows(page, xn, TNODE_R, EXT_R, p.out, 0))
  }
@@ -584,8 +624,9 @@ export function buildTransitionGraph(data, pageTree) {
  const pairs = aggregatePairs(internal)
  const { edges, flows } = buildInternalEdges(pairs, byPath)

-  // Tight bounding box of the actual page nodes; internal edges and arcs
-  // stay within the node circles, so node bounds plus radius suffice.
+  // Tight bounding box of the actual page nodes; family ring arcs can sweep
+  // outside the node circle (e.g. a large arc between two siblings on the
+  // left side reaching around the right), so their geometry is included too.
  // External nodes extend the box below.
  const pad = 16
  const xs = nodes.map((n) => n.x)
@@ -596,6 +637,14 @@ export function buildTransitionGraph(data, pageTree) {
    x1: Math.max(...xs) + TNODE_R + pad,
    y1: Math.max(...ys) + TNODE_R + pad,
  }
+  for (const arc of arcs) {
+    if (arc.a0 == null) continue
+    const b = arcBounds(arc.r, arc.a0, arc.a1)
+    bounds.x0 = Math.min(bounds.x0, b.x0)
+    bounds.y0 = Math.min(bounds.y0, b.y0)
+    bounds.x1 = Math.max(bounds.x1, b.x1)
+    bounds.y1 = Math.max(bounds.y1, b.y1)
+  }

  const ext = buildExternal(external, byPath, radius, bounds)
  for (const xn of ext.extNodes) {
@@ -425,7 +425,11 @@ import "overlayscrollbars/overlayscrollbars.css";
        if (!res.ok || !type.includes("text/html")) throw new Error("not a page");
        // Reflect any redirect the server issued.
        if (res.redirected) finalUrl = res.url;
-        doc = new DOMParser().parseFromString(await res.text(), "text/html");
+        const html = await res.text();
+        // Populate the cache too, or the post-swap preload (which includes
+        // location.pathname) would fetch the very page we just loaded again.
+        pageCache.set(new URL(finalUrl, location.href).pathname, html);
+        doc = new DOMParser().parseFromString(html, "text/html");
      } catch {
        location.href = url; // fall back to a normal navigation
        return false;
@@ -471,7 +475,9 @@ import "overlayscrollbars/overlayscrollbars.css";
      runScripts(document.getElementById("page-banner"));
      runScripts(document.getElementById("main"));
      applyEffects();
-      mountAnalytics(document);
+      // The fetched doc carries the analytics meta; the live document's
+      // <head> is never swapped, so querying it would never find the entry.
+      mountAnalytics(doc);
    };
    // Rotating cube page transition (see the FRAGILE block in pagerite.css);
    // mirrored when navigating back through history. Navigation within the
@@ -530,9 +536,9 @@ import "overlayscrollbars/overlayscrollbars.css";
    if (!a || a.target || a.hasAttribute("download")) return;
    const url = new URL(a.href, location.href);
    if (url.origin !== location.origin) {
-      // External link: the browser navigates; just record the exit (https
-      // origins only, stripped to the origin part server-side anyway).
-      if (url.protocol === "https:") ping(url.origin);
+      // External link: the browser navigates; record the full https URL so
+      // different links to the same domain stay distinct in analytics.
+      if (url.protocol === "https:") ping(url.href);
      return;
    }
    // Same-page anchor links (footnotes etc.): let the browser handle them
@@ -11,7 +11,7 @@
 */

 export default function fastapiVue({ paths = ["/api"] } = {}) {
-  const backendUrl = process.env.PAGERITE_BACKEND_URL || "http://localhost:3200"
+  const backendUrl = process.env.PAGERITE_BACKEND_URL || "http://localhost:8210"

  // Build proxy configuration for each path
  const proxy = {}
@@ -1,14 +1,74 @@
-# auto-upgrade@fastapi-vue-setup - remove this if you modify this file
 """Command-line entry point for running the backend server."""

 import argparse
+import gzip
 import os
+import sys
+from datetime import date
+from pathlib import Path

+import httpx
 from fastapi_vue import server

-DEFAULT_PORT = 3100
+DEFAULT_PORT = 8100
 DEVMODE = os.getenv("PAGERITE_DEV") == "1"

+# Repository root (pagerite/__main__.py -> ..), where the MMDB lives.
+_REPO_ROOT = Path(__file__).resolve().parent.parent
+
+DBIP_URL = "https://download.db-ip.com/free/dbip-city-lite-{month}.mmdb.gz"
+
+
+def _download_dbip() -> None:
+    """Download the latest dbip-city-lite MMDB if ours is missing or older."""
+    today = date.today()
+    months = [f"{today:%Y-%m}"]
+    # The current month's file may not be published yet; fall back to last month.
+    prev = (today.replace(day=1) - date.resolution).replace(day=1)
+    months.append(f"{prev:%Y-%m}")
+
+    existing = sorted(
+        p.stem.removeprefix("dbip-city-lite-").removesuffix(".mmdb")
+        for p in _REPO_ROOT.glob("dbip-city-lite-*.mmdb*")
+    )
+    if existing and existing[-1] >= months[0]:
+        print(f"pagerite: DB-IP database is current ({existing[-1]}), skipping download")
+        return
+
+    for month in months:
+        url = DBIP_URL.format(month=month)
+        target = _REPO_ROOT / f"dbip-city-lite-{month}.mmdb.gz"
+        tmp = target.with_suffix(".mmdb.gz.tmp")
+        print(f"pagerite: downloading {url}")
+        try:
+            with httpx.stream("GET", url, follow_redirects=True, timeout=120) as r:
+                if r.status_code == 404:
+                    continue
+                r.raise_for_status()
+                with open(tmp, "wb") as f:
+                    for chunk in r.iter_bytes():
+                        f.write(chunk)
+        except httpx.HTTPError as e:
+            print(f"pagerite: DB-IP download failed: {e}", file=sys.stderr)
+            tmp.unlink(missing_ok=True)
+            continue
+        # Verify it is actually gzip data before installing it.
+        try:
+            with gzip.open(tmp, "rb") as f:
+                f.read(1)
+        except OSError:
+            print(f"pagerite: DB-IP download for {month} was not valid gzip", file=sys.stderr)
+            tmp.unlink(missing_ok=True)
+            continue
+        os.replace(tmp, target)
+        # Drop older databases so the app never picks up a stale one.
+        for old in _REPO_ROOT.glob("dbip-city-lite-*.mmdb*"):
+            if old.name != target.name:
+                old.unlink()
+        print(f"pagerite: DB-IP database updated to {target.name}")
+        return
+    print("pagerite: could not download a DB-IP database", file=sys.stderr)
+

 def main() -> None:
    """Run the backend server with optional arguments."""
@@ -19,7 +79,14 @@ def main() -> None:
        action="append",
        help=(f"Endpoint (default: localhost:{DEFAULT_PORT})."),
    )
+    parser.add_argument(
+        "--dbip",
+        action="store_true",
+        help="Download/update the DB-IP city lite database before starting.",
+    )
    args = parser.parse_args()
+    if args.dbip:
+        _download_dbip()
    dev = {"reload": True, "reload_dirs": ["pagerite"]} if DEVMODE else {}
    server.run(
        "pagerite.app:app",
@@ -17,6 +17,8 @@ rewritten atomically on every recorded event.
 import os
 import re
 import tempfile
+from collections.abc import Callable
+from contextlib import suppress
 from datetime import UTC, datetime, timedelta
 from pathlib import Path
 from urllib.parse import parse_qs, urlparse
@@ -50,7 +52,7 @@ def _compact_user_agent(ua: str) -> str:
 class Visit(msgspec.Struct, omit_defaults=True):
    """One visit: the initial-load data plus everything seen afterwards.

-    ``trail`` holds page paths and external exit origins in first-seen
+    ``trail`` holds page paths and external exit URLs in first-seen
    order; re-visiting an already seen page does not append. The entry
    page itself is in ``entry``, not in the trail.
    """
@@ -67,7 +69,10 @@ class Visit(msgspec.Struct, omit_defaults=True):
    #: First Accept-Language tag, lowercased (e.g. "en-us").
    lang: str = ""
    #: Two-letter region subtag derived from ``lang`` (e.g. "US"), or "".
+    #: Overwritten by the DB-IP geoip lookup when a database is available.
    country: str = ""
+    #: City name from the DB-IP geoip lookup, or "".
+    city: str = ""
    #: Raw User-Agent header from the initial ping.
    ua: str = ""
    #: Compact display form of ``ua`` (browser/OS/device) when parsable.
@@ -124,6 +129,17 @@ def _origin(url: str) -> str | None:
    return f"https://{parsed.netloc}"


+def _external_target(url: str) -> str | None:
+    """A valid https URL (origin or full page), else None."""
+    try:
+        parsed = urlparse(url)
+    except ValueError:
+        return None
+    if parsed.scheme != "https" or not parsed.netloc:
+        return None
+    return url
+
+
 _SEGMENT = re.compile(r"[a-z0-9][a-z0-9_-]*")


@@ -196,6 +212,23 @@ class Store:
        #: Document GETs that have not yet been matched by a ping.  Kept
        #: in RAM only; expired entries are written to ``data.crawlers``.
        self.pending_crawlers: list[CrawlerHit] = []
+        #: Callables to notify when persisted data changes.  Registered by the
+        #: analytics WebSocket broadcaster.
+        self._on_change: list[Callable[[], None]] = []
+
+    def subscribe(self, callback: Callable[[], None]) -> None:
+        """Register a callback to be called after every persisted change."""
+        if callback not in self._on_change:
+            self._on_change.append(callback)
+
+    def unsubscribe(self, callback: Callable[[], None]) -> None:
+        """Remove a previously registered change callback."""
+        with suppress(ValueError):
+            self._on_change.remove(callback)
+
+    def _notify(self) -> None:
+        for callback in self._on_change:
+            callback()

    def _save(self) -> None:
        """Rewrite the JSON file atomically (temp file + rename)."""
@@ -208,6 +241,8 @@ class Store:
            os.replace(tmp, self.path)
        except OSError:
            pass  # analytics must never break page serving
+        else:
+            self._notify()

    def _flush_crawlers(self, now: datetime | None = None) -> None:
        """Move expired pending crawler hits into persistent ``data.crawlers``."""
@@ -268,6 +303,7 @@ class Store:
        *,
        host: str = "",
        country: str = "",
+        city: str = "",
    ) -> None:
        """Fill in host/geoip fields on an existing visit after async lookups."""
        if index < 0 or index >= len(self.data.visits):
@@ -280,6 +316,9 @@ class Store:
        if country:
            visit.country = country
            changed = True
+        if city:
+            visit.city = city
+            changed = True
        if changed:
            self._save()

@@ -336,9 +375,9 @@ class Store:
    ) -> int | None:
        """Record a client navigation ping ({from, to} from pagerite.js).

-        ``to`` is an internal path ("/...") or an https origin for exit
-        links; anything else is ignored. The transition is always counted;
-        the trail only grows on first sight of a page within the visit.
+        ``to`` is an internal path ("/...") or an https URL for exit links;
+        anything else is ignored. The transition is always counted; the trail
+        only grows on first sight of a page within the visit.
        A ping with no known session starts a fresh visit, consuming the
        referer and UTM tags stashed by the document GET if there are any.

@@ -354,8 +393,8 @@ class Store:
        if to.startswith("/") and not to.startswith("//"):
            target = _internal_path(to) or ""
        else:
-            target = _origin(to) or ""
-        if not target or (not to.startswith("/") and target != to):
+            target = _external_target(to) or ""
+        if not target:
            return None
        key = (ip, ua)
        index = self.sessions.get(key)
@@ -57,6 +57,10 @@ ANALYTICS_PATH = Path(
 )
 analytics_store = analytics.Store(ANALYTICS_PATH)

+# Live WebSocket clients for the analytics stream.
+_analytics_ws_clients: set[WebSocket] = set()
+_analytics_broadcast_task: asyncio.Task | None = None
+

 # Repository root from this file's location (pagerite/app.py -> ..).
 _REPO_ROOT = Path(__file__).resolve().parent.parent
@@ -121,6 +125,18 @@ class GeoIP:
            pass
        return ""

+    def city(self, ip: str) -> str:
+        """City name for ``ip``, or "" when unavailable."""
+        if not ip or self._reader is None:
+            return ""
+        try:
+            rec = self._reader.get(ip)
+            if rec:
+                return (rec.get("city") or {}).get("names", {}).get("en", "")
+        except Exception:
+            pass
+        return ""
+

 _geoip = GeoIP()

@@ -231,7 +247,9 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
    # Decompress/open the DB-IP MMDB once at startup.  Lookups are then
    # read-only and safe to run in background ``to_thread`` workers.
    await asyncio.to_thread(_geoip._load)
+    analytics_store.subscribe(_schedule_analytics_broadcast)
    yield
+    analytics_store.unsubscribe(_schedule_analytics_broadcast)
    await kanta.close()


@@ -615,13 +633,50 @@ async def _geoip_country(ip: str) -> str:
    return await asyncio.to_thread(_geoip.country, ip)


+async def _geoip_city(ip: str) -> str:
+    """Async wrapper around the DB-IP MMDB city lookup."""
+    return await asyncio.to_thread(_geoip.city, ip)
+
+
 async def _enrich_visit(index: int, ip: str) -> None:
    """Run non-blocking reverse-DNS and geoip enrichment for a new visit."""
    if not ip:
        return
    host = await _lookup_host(ip)
    country = await _geoip_country(ip)
-    analytics_store.enrich_visit(index, host=host, country=country)
+    city = await _geoip_city(ip)
+    analytics_store.enrich_visit(index, host=host, country=country, city=city)
+
+
+async def _broadcast_analytics() -> None:
+    """Send the current analytics snapshot to every connected WS client."""
+    if not _analytics_ws_clients:
+        return
+    payload = msgspec.json.encode(analytics_store.data).decode()
+    closed = set()
+    for ws in _analytics_ws_clients:
+        try:
+            await ws.send_text(payload)
+        except Exception:
+            closed.add(ws)
+    for ws in closed:
+        _analytics_ws_clients.discard(ws)
+
+
+async def _debounced_analytics_broadcast() -> None:
+    """Wait briefly, then broadcast the latest snapshot once."""
+    await asyncio.sleep(0.2)
+    await _broadcast_analytics()
+
+
+def _schedule_analytics_broadcast() -> None:
+    """Schedule a single debounced broadcast, ignoring duplicate triggers."""
+    global _analytics_broadcast_task
+    if _analytics_broadcast_task is not None and not _analytics_broadcast_task.done():
+        return
+    _analytics_broadcast_task = asyncio.get_running_loop().create_task(
+        _debounced_analytics_broadcast()
+    )


 class AnalyticsPing(BaseModel):
@@ -635,7 +690,7 @@ class AnalyticsPing(BaseModel):
 async def analytics_page(request: Request) -> HTMLResponse:
    """Render the analytics viewer as a normal site page at /_a.

-    The page itself is public, but the data endpoint (/_api/analytics) stays
+    The page itself is public, but the data stream (/_api/ws/analytics) stays
    admin-gated like the rest of /_api, so only authorized users see the
    statistics; others get the viewer with a "could not be loaded" message.
    """
@@ -727,16 +782,23 @@ def _check_reserved(path: str) -> None:
        )


-@app.get("/_api/analytics")
-async def get_analytics() -> Response:
-    """The collected visit analytics as JSON (see docs/analytics.md).
+@app.websocket("/_api/ws/analytics")
+async def analytics_websocket(ws: WebSocket) -> None:
+    """Stream the analytics snapshot, then push updates as they happen.

    Admin-only via the /_api forward-auth gate, like every management
    endpoint. Powers the analytics viewer rendered at /_a.
    """
-    return Response(
-        msgspec.json.encode(analytics_store.data), media_type="application/json"
-    )
+    await ws.accept()
+    await ws.send_text(msgspec.json.encode(analytics_store.data).decode())
+    _analytics_ws_clients.add(ws)
+    try:
+        while True:
+            await ws.receive_text()
+    except Exception:
+        pass
+    finally:
+        _analytics_ws_clients.discard(ws)


@app.websocket("/_api/ws/editor")
@@ -29,6 +29,7 @@ Where to go next:
 - The [docs](/docs/editing) section explains how to edit this site and shows every supported Markdown feature, source and result side by side.
 - The [showcase](/showcase/gallery) section shows what finished pages can look like: image positioning, banners, a long read.
 - Click the 🖊️ pen on any page to open the editor, and the ⚙️ pen for site settings and the structure tree.
+- Elsewhere on the web: [![xkcd 927: Standards](https://imgs.xkcd.com/comics/standards.png "xkcd 927: Standards"){width=240}](https://xkcd.com/927/) — a cautionary tale about adding one more standard.

 ![Abstract waves](waves.svg "Generated SVG artwork, attached to this page"){width=420}

@@ -68,15 +69,15 @@ Every feature below is shown twice: first the Markdown source, then how it rende
 ### A subsection

 *Emphasis*, **strong**, ~~strikethrough~~, `inline code`, and a
-[link to the front page](/). Plain URLs become links automatically:
-https://example.com — and a hard line break
+[link to the front page](/). An image that links to its page:
+[![xkcd 1179: ISO 8601](https://imgs.xkcd.com/comics/iso_8601.png "xkcd 1179: ISO 8601"){width=240}](https://xkcd.com/1179/) — and a hard line break
 is just a newline.
 ```

 ## A section heading
 ### A subsection

-*Emphasis*, **strong**, ~~strikethrough~~, `inline code`, and a [link to the front page](/). Plain URLs become links automatically: https://example.com — and a hard line break
+*Emphasis*, **strong**, ~~strikethrough~~, `inline code`, and a [link to the front page](/). An image that links to its page: [![xkcd 1179: ISO 8601](https://imgs.xkcd.com/comics/iso_8601.png "xkcd 1179: ISO 8601"){width=240}](https://xkcd.com/1179/) — and a hard line break
 is just a newline.

 ## Lists and quotes
@@ -354,6 +355,7 @@ This site runs on **Pagerite**: FastAPI + html5tagger + kanta, with content writ
 - [How to edit this site](/docs/editing)
 - [Markdown features](/docs/markdown/basics)
 - [The showcase](/showcase/gallery)
+- [![xkcd 2347: Dependency](https://imgs.xkcd.com/comics/dependency.png "xkcd 2347: Dependency"){width=240}](https://xkcd.com/2347/) — a small comic about small dependencies

 *Replace this page with whatever your site is about.*
 """
@@ -20,6 +20,7 @@ dependencies = [
    "fastapi-vue>=1.3.1",
    "fastapi[standard]>=0.141.1",
    "html5tagger>=2.0.0",
+    "httpx>=0.28.1",
    "kanta>=0.8.1",
    "markdown-it-py>=4.2.0",
    "maxminddb>=3.1.1",
@@ -35,9 +36,7 @@ pagerite = "pagerite.__main__:main"
 Repository = "https://git.zi.fi/LeoVasanko/pagerite"

 [dependency-groups]
-dev = [
-    "httpx>=0.28.1",
-]
+dev = []

 [tool.hatch.version]
 source = "vcs"
@@ -19,8 +19,8 @@ from devutil import (
    setup_vite,
 )

-DEFAULT_VITE_PORT = 3100
-DEFAULT_DEV_PORT = 3200
+DEFAULT_VITE_PORT = 8200
+DEFAULT_DEV_PORT = 8210
 HEALTH = "/?from=devserver.py"


@@ -10,9 +10,12 @@

 The script drives a real Chromium browser with Playwright, clicking visible
 internal links so the site's own analytics JavaScript records normal visits
-(POST /_a).  Browser sessions and crawler GETs send a small rotating pool of
-real public IPs in X-Forwarded-For, so the backend can reverse-DNS and GeoIP
-them instead of seeing every hit as 127.0.0.1.
+(POST /_a).  Most browser sessions enter the site with a cross-origin
+``Referer: https://somedomain.com/`` header, and outbound links found on the
+page are followed to real external sites (ending the session).  Browser
+sessions and crawler GETs send a small rotating pool of real public IPs in
+X-Forwarded-For, so the backend can reverse-DNS and GeoIP them instead of seeing
+every hit as 127.0.0.1.

 Sessions start with a Poisson inter-arrival delay (``--arrival-rate``) to
 spread traffic out a little, while still keeping the overall run fast.
@@ -125,6 +128,30 @@ def _source_ip(index: int) -> str:
    return SOURCE_IPS[index % len(SOURCE_IPS)]


+def _normalize_url(url: str) -> str:
+    """Return a usable base URL, adding missing scheme/host/port parts.
+
+    - bare ``:PORT`` becomes ``http://localhost:PORT``
+    - missing scheme becomes ``http://``
+    - otherwise returned as-is
+
+    Raises ``ValueError`` when the result is not a valid http(s) URL.
+    """
+    raw = url.strip()
+    if not raw:
+        raise ValueError("empty URL")
+    if raw.startswith(":"):
+        raw = f"http://localhost{raw}"
+    elif raw.isdigit():
+        raw = f"http://localhost:{raw}"
+    elif not raw.startswith(("http://", "https://")):
+        raw = f"http://{raw}"
+    parsed = urlparse(raw)
+    if parsed.scheme not in ("http", "https") or not parsed.netloc:
+        raise ValueError(f"invalid URL: {url!r}")
+    return raw
+
+
 def _poisson_wait(rate: float) -> float:
    """Return an exponential inter-arrival time for the given Poisson rate."""
    if rate <= 0:
@@ -132,31 +159,39 @@ def _poisson_wait(rate: float) -> float:
    return random.expovariate(rate)


-def _collect_links(page: Any) -> list[dict[str, Any]]:
-    """Return internal links from the current page, excluding the current page."""
+def _collect_links(page: Any, include_external: bool = False) -> list[dict[str, Any]]:
+    """Return links from the current page, excluding the current page.
+
+    Internal links stay on the site; external links are real https URLs found
+    in the page content and are marked with ``external: true``.
+    """
    return page.evaluate(
-        """() => {
+        """(includeExternal) => {
            const loc = new URL(location.href);
-            return Array.from(document.querySelectorAll('a[href]'))
-                .filter(a => {
-                    try {
-                        const u = new URL(a.href);
-                        return u.origin === loc.origin
-                            && !u.pathname.startsWith('/_')
-                            && !u.pathname.startsWith('/auth')
-                            && u.pathname !== '/favicon.ico'
-                            && u.pathname !== loc.pathname;
-                    } catch { return false; }
-                })
-                .map(a => {
+            const out = [];
+            for (const a of document.querySelectorAll('a[href]')) {
+                try {
+                    const u = new URL(a.href);
                    const rect = a.getBoundingClientRect();
-                    return {
+                    const item = {
                        href: a.href,
                        text: (a.innerText || a.title || '').trim().slice(0, 60),
                        visible: !!(rect.width && rect.height && rect.top < window.innerHeight && rect.bottom > 0),
                    };
-                });
-        }"""
+                    if (u.origin === loc.origin
+                            && !u.pathname.startsWith('/_')
+                            && !u.pathname.startsWith('/auth')
+                            && u.pathname !== '/favicon.ico'
+                            && u.pathname !== loc.pathname) {
+                        out.push(item);
+                    } else if (includeExternal && u.protocol === 'https:' && u.origin !== loc.origin) {
+                        out.push({ ...item, external: true });
+                    }
+                } catch { /* ignore malformed hrefs */ }
+            }
+            return out;
+        }""",
+        include_external,
    )


@@ -201,6 +236,8 @@ def _run_browser_session(
    stay: tuple[float, float],
    headless: bool,
    fake_ip: str,
+    referer_rate: float,
+    include_external: bool = True,
 ) -> dict[str, Any]:
    from playwright.sync_api import sync_playwright

@@ -212,13 +249,17 @@ def _run_browser_session(
                headless=headless,
                args=["--no-sandbox", "--disable-dev-shm-usage"],
            )
+            extra_headers = {
+                "X-Forwarded-For": fake_ip,
+                "Accept-Language": profile.accept_language,
+            }
+            # Most sessions arrive from an external origin; some are direct.
+            if random.random() < referer_rate:
+                extra_headers["Referer"] = "https://somedomain.com/"
            context = browser.new_context(
                user_agent=profile.user_agent,
                viewport={"width": profile.viewport[0], "height": profile.viewport[1]},
-                extra_http_headers={
-                    "X-Forwarded-For": fake_ip,
-                    "Accept-Language": profile.accept_language,
-                },
+                extra_http_headers=extra_headers,
            )
            page = context.new_page()
            entry = random.choice(paths) if paths else "/"
@@ -227,7 +268,7 @@ def _run_browser_session(

            for _ in range(max_clicks):
                _sleep(random.uniform(*stay) / 2, 0.3)
-                links = _collect_links(page)
+                links = _collect_links(page, include_external)
                visible = [item for item in links if item.get("visible")]
                if not visible:
                    visible = links
@@ -242,6 +283,12 @@ def _run_browser_session(
                        ok = _click_link(page, alt)
                    if not ok:
                        break
+                if link.get("external"):
+                    # Outbound navigation: the analytics exit ping is already
+                    # in flight. Record the external URL and end the session.
+                    trail.append(page.url)
+                    _sleep(0.5, 0.2)
+                    break
                page.wait_for_load_state("networkidle")
                trail.append(page.url)
                _sleep(random.uniform(*stay), 0.5)
@@ -295,7 +342,14 @@ def _parse_args(argv: Sequence[str] | None) -> argparse.Namespace:
        description="Generate fake traffic for a Pagerite site.",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
-    parser.add_argument("url", help="Base URL of the Pagerite site")
+    parser.add_argument(
+        "url",
+        nargs="?",
+        default="http://localhost:8200",
+        help="Base URL of the Pagerite site (default: http://localhost:8200). "
+             "A bare :PORT or PORT is treated as http://localhost:PORT; a "
+             "missing scheme defaults to http://.",
+    )
    parser.add_argument(
        "-b",
        "--browsers",
@@ -332,6 +386,18 @@ def _parse_args(argv: Sequence[str] | None) -> argparse.Namespace:
        default=1.0,
        help="Average arrivals per second (Poisson). 0 disables inter-arrival waits",
    )
+    parser.add_argument(
+        "--referer-rate",
+        type=float,
+        default=0.75,
+        help="Share of browser sessions that arrive with a cross-origin Referer",
+    )
+    parser.add_argument(
+        "--external-links",
+        action=argparse.BooleanOptionalAction,
+        default=True,
+        help="Include real outbound links in random navigation",
+    )
    parser.add_argument("--seed", type=int, default=None, help="Random seed")
    parser.add_argument("-v", "--verbose", action="store_true", help="Debug logging")
    return parser.parse_args(argv)
@@ -342,8 +408,13 @@ def main(argv: Sequence[str] | None = None) -> int:
    if args.verbose:
        logger.setLevel(logging.DEBUG)

+    try:
+        base = _normalize_url(args.url).rstrip("/")
+    except ValueError as exc:
+        logger.error("%s", exc)
+        return 2
+
    random.seed(args.seed)
-    base = args.url.rstrip("/")

    # Discover content paths from the public page tree if we can.
    paths: list[str] = []
@@ -389,6 +460,8 @@ def main(argv: Sequence[str] | None = None) -> int:
            (args.stay[0], args.stay[1]),
            args.headless,
            fake_ip,
+            args.referer_rate,
+            args.external_links,
        )
        results.append(result)
        logger.debug("  trail: %s", result.get("trail", []))
Author	SHA1	Message	Date
LeoVasanko	1a479ceb24	Add --dbip CLI flag to auto-download/update the DB-IP MMDB database. Downloads the latest dbip-city-lite-YYYY-MM.mmdb.gz before starting the server, skipping when the local database is current, falling back to the previous month on 404, and removing older databases after an update. Promotes httpx to a runtime dependency.	2026-08-21 03:09:04 +00:00
LeoVasanko	ff553d018a	Default scheme, host and port for fake_traffic script.	2026-08-21 02:52:53 +00:00
LeoVasanko	c807d48a13	Add more external content in seed data.	2026-08-21 02:50:36 +00:00
LeoVasanko	9c383c1c8b	Change default port mapping to 8100/8200/8210 (prod/vite/dev). Vite gets different port to avoid caching problems when switching between it and prod.	2026-08-21 02:49:18 +00:00
LeoVasanko	462e995adc	Add external link (referer/outgoing) display on connection graph.	2026-08-21 02:44:54 +00:00
LeoVasanko	ea069b98da	Fix analytics app not mounting on fetch-navigation to /_a load() queried the live document for the pagerite:analytics-src meta, but the swap never touches <head> — the meta only exists in the fetched doc, so the app never mounted unless /_a was loaded directly. Also cache the fetched HTML so the post-swap preload doesn't re-GET the page we just navigated to.	2026-08-21 01:53:39 +00:00
LeoVasanko	deb5419c47	analytics improvements: - keep visitor charts y-axis minimum range at 10 - keep 'all' chart x-axis minimum span at 30 days - group crawler hits by (ip, ua) and list top pages visited, show crawler page load counts as N× prefix - store and display geoip city, keep geoip country overwrite - stream live updates over WebSocket /_api/ws/analytics - include family ring arcs in transition map crop bounds - remove top UA summary, limit crawlers to 10 and visits to 20 - human-readable relative timestamps with UTC tooltip	2026-08-21 01:36:58 +00:00