Slightly prettier analytics URL

docs/analytics.md

@@ -5,8 +5,9 @@ Struct dumped to disk — separate from the kanta content database, path from
 `PAGERITE_ANALYTICS` (default: the database path with `.kantadb` replaced by
 `.analytics.json`, e.g. `pagerite.analytics.json`).
 
-- `pagerite/analytics.py` — data model (`Analytics`, `Visit`) and the `Store`
-  (in-memory data + session map, atomic JSON persistence).
+- `pagerite/analytics.py` — data model (`Analytics`, `Client`, `Visit`,
+  `CrawlerHit`, `AbuseHit`) and the `Store` (in-memory data + session map,
+  atomic JSON persistence).
 - `pagerite/app.py` — entry-referer stashing in `show_page` (`_track_entry`),
   the `POST /_a` ping endpoint, and `WebSocket /_api/ws/analytics`
   (admin-gated like every `/_api` endpoint).
@@ -38,80 +39,127 @@ The client (`pagerite.js`) POSTs fire-and-forget pings to `/_a` with
   links are stored by full URL so several links to the same domain remain
   distinct.
 - **Excluded**: back/forward (popstate) navigations, navigation involving
-  the analytics page itself (`/_a`), and everything while the user is known to
-  be an admin *and SSO is actually in use* — with no auth proxy (dev/test)
-  "admin" is everyone's state, so the gate is off and everything is recorded —
-  or has the editor open (`body.editing`). Admin noise, not visits.
+  the analytics page itself (`/_a`), and everything while the user has the
+  editor open (`body.editing`). Admin noise, not visits.
+- **Admins**: when SSO is in use and the session is known to be an admin,
+  the client still pings but adds `hide=1`. The server then records
+  nothing — and if the same client session already had a visit from before
+  logging in, that visit is removed from the JSON along with the counts
+  recorded when it was created (site visit, entry view, entry transition).
+  Views/transitions logged by later pings inside such a visit lack
+  per-event timestamps and are left as-is. With no auth proxy (dev/test)
+  "admin" is everyone's state, so `hide` stays 0 and everything is recorded.
 - The server validates `to`: internal paths must be valid slug paths
   ("/" or `[a-z0-9_-]` segments), external ones are re-derived to the
   https origin and accepted only when the client sent exactly that.
-- The initial ping also records the visitor's `User-Agent` and
-  `Accept-Language` headers. The first `Accept-Language` tag is stored as
-  `lang` (e.g. `en-us`) and its region subtag, if present, is stored as
-  an initial `country` (e.g. `US`).
-- The visitor IP is stored.  A reverse-DNS lookup is attempted for each new
-  visit and the result, when available, is cached in RAM and stored as
-  `host`; local/reserved/multicast addresses are skipped.
-- If a DB-IP MMDB file (`dbip-*.mmdb` or `dbip-*.mmdb.gz`) is present in the
-  repository root, it is loaded at startup and used to look up a more accurate
-  `country`.  The MMDB lookup and the reverse-DNS lookup run in background
-  tasks after the visit is stored, so the `/ _a` response is never delayed.
-  The decompressed `dbip-*.mmdb` file is kept in the repository root and
-  ignored by git.  The CLI flag `--dbip` (`uv run pagerite --dbip`) downloads
-  the latest `dbip-city-lite-YYYY-MM.mmdb.gz` from DB-IP before the server
-  starts, skipping the download when the local database is already current and
+- **Client records**: the visitor's IP (IPv4 or IPv6 /64 network), raw
+  `User-Agent` and extracted `Accept-Language` tag are hashed with blake3;
+  the first 6 bytes identify a shared `Client` record.  The `Client` stores
+  the full IP, `User-Agent`, compact `ua_pretty`, `lang`, initial
+  `country` from the language-region subtag, and asynchronously-filled
+  `country`/`city` from DB-IP geoip plus reverse-DNS `host`.  Visits,
+  crawler hits and abuse hits all reference this record by its hash, so
+  client metadata is stored once instead of repeated per event.
+- The visitor IP is stored in the `Client`.  A reverse-DNS lookup is
+  attempted for each new client and the result, when available, is stored as
+  `host`; local/reserved/multicast addresses are skipped.  If a DB-IP MMDB
+  file (`dbip-*.mmdb` or `dbip-*.mmdb.gz`) is present in the repository
+  root, it is loaded at startup and used to look up `country`/`city`.  These
+  lookups run in background tasks after the event is stored, so the `/_a`
+  response is never delayed.  The decompressed `dbip-*.mmdb` file is kept in
+  the repository root and ignored by git.  The CLI flag `--dbip`
+  (`uv run pagerite --dbip`) downloads the latest
+  `dbip-city-lite-YYYY-MM.mmdb.gz` from DB-IP before the server starts,
+  skipping the download when the local database is already current and
   removing older versions after an update; without the flag only an existing
   file is used.
 - **Crawler hits**: every document GET is queued in RAM as a pending crawler
-  hit.  If a ping from the same (IP, User-Agent) pair arrives within 10
-  seconds the hit is discarded; otherwise it is written to `crawlers`.
-  Crawlers do not count as visits or views.  In the analytics viewer, crawler
-  hits are grouped by the same (IP, User-Agent) pair and shown as a trail of
-  internal pages that crawler visited; the crawler table lists the most active
-  crawlers first rather than the most recent hits.
+  hit.  If a ping from the same client arrives within 10 seconds the hit is
+  discarded; otherwise it is written to `crawlers`.  Crawlers do not count as
+  visits or views.  The `Accept-Language` header is stored on the shared
+  `Client` immediately; reverse-DNS host names and DB-IP geoip
+  country/city are filled in asynchronously, just like for real visits.  In
+  the analytics viewer, crawler hits are grouped by client hash and shown as
+  a trail of internal pages that crawler visited; the crawler table lists
+  the most active crawlers first rather than the most recent hits.
+- **Abuse (scanner) hits**: a 404 for a telltale path — any URL segment
+  starting with a dot (`/.env`, `/.git/config`) or ending in `.php` —
+  classifies the source IP as abuse immediately, and ten plain 404s from one
+  IP do too.  Classification reclassifies history: all earlier crawler hits
+  from that IP (persisted and pending) move to the `abuse` list, so a
+  random-UA scanner no longer pollutes the crawler stats of the legitimate
+  bot it impersonates.  Once classified, every document GET and 404 from the
+  IP is recorded as an abuse hit with the full request path (query string
+  included), and its pings are ignored.  The classified IP set (`abuse_ips`)
+  is persisted in the JSON file; the plain-404 counters are RAM-only.  In the
+  viewer, abuse hits are grouped by IP (never by client/UA — scanners
+  randomize theirs) in a separate "Abuse" table.  Identical paths are
+  collapsed into one entry with their hit count; flagged paths that
+  triggered classification are lifted to the top, followed by other 404s and
+  then document GETs from the abuser.  Raw User-Agent strings are shown one
+  per line with their occurrence counts, and the full lists are click-to-copy.
 
 ## Visits and sessions
 
-There are no cookies. A visit is tied together by the (IP, User-Agent) pair
-(IP from the first `X-Forwarded-For` hop — we sit behind a proxy — else the
-direct peer): the first ping from a pair starts a new visit, subsequent
-pings extend it. Pings arriving with no known session (server restart)
-start a fresh visit from the first ping — treated as missing data rather
-than dropped. The (IP, UA) → visit map and the IP → entry-referer/UTM
-tables are in-memory only, but the IP and any resolvable reverse-DNS host
-name are stored on the `Visit` record itself.
+There are no cookies. A visit is tied together by a client hash — the first
+6 bytes of a blake3 digest over the prettified IP (IPv4 unchanged, IPv6
+/64 network), the raw `User-Agent` string and the extracted
+`Accept-Language` tag.  The first ping from a client hash starts a new
+visit; subsequent pings extend it.  Pings arriving with no known session
+(server restart) start a fresh visit from the first ping — treated as
+missing data rather than dropped.  The client-hash → visit map and the IP →
+entry-referer/UTM tables are in-memory only; client metadata is stored in
+`Analytics.clients` keyed by the client hash.
+
+Each `Client` record:
+
+- `ip` — visitor IP address (first `X-Forwarded-For` hop, or direct peer),
+- `host` — reverse-DNS host name for `ip` when resolvable, else `""`,
+- `lang` — first `Accept-Language` tag, lowercased (e.g. `"en-us"`),
+- `country` — two-letter country code.  Initially derived from the
+  `Accept-Language` region subtag, but overwritten by the DB-IP MMDB result
+  when a database is available,
+- `city` — city name from the DB-IP MMDB lookup, when available,
+- `ua` — raw `User-Agent` string,
+- `ua_pretty` — compact display form of the UA (browser/OS/device) when
+  parsable, otherwise the raw string.
 
 Each `Visit` record:
 
 - `start` — timestamp of the first event,
 - `entry` — first page (path) seen,
 - `referer` — external https origin of the initial load, `""` for direct,
-- `ip` — visitor IP address (first `X-Forwarded-For` hop, or direct peer),
-- `host` — reverse-DNS host name for `ip` when resolvable, else `""`,
+- `client` — 6-byte blake3 hash referencing `Analytics.clients`,
 - `trail` — everything seen afterwards in first-seen order: page paths and
   external exit URLs. Re-visiting an already seen page (incl. the entry)
   does not append.
-- `lang` — first `Accept-Language` tag, lowercased (e.g. `en-us`),
-- `country` — two-letter country code.  Initially derived from the
-  `Accept-Language` region subtag, but overwritten by the DB-IP MMDB result
-  when a database is available,
-- `city` — city name from the DB-IP MMDB lookup, when available,
-- `ua` — raw `User-Agent` string from the initial ping,
-- `ua_pretty` — compact display form of the UA (browser/OS/device) when
-  parsable, otherwise the raw string,
 - `utm` — `utm_*` query parameters from the landing URL, as a dict.
+- `read` — active reading time per path (seconds), keyed by path.
 
 Each `CrawlerHit` record:
 
 - `start` — timestamp of the document GET,
 - `entry` — page path requested,
-- `ip` — IP address,
-- `ua` — raw `User-Agent` header,
-- `ua_pretty` — compact display form of the UA when parsable,
+- `client` — 6-byte blake3 hash referencing `Analytics.clients`,
 - `referer` — external https origin of the request, `""` for direct/none,
 - `query` — raw query string of the request.
 
-Crawler hits are grouped by User-Agent in the analytics viewer.
+Each `AbuseHit` record:
+
+- `start` — timestamp of the request,
+- `path` — full request path including the query string (e.g. `/.env?x=1`),
+- `client` — 6-byte blake3 hash referencing `Analytics.clients`,
+- `flag` — true for the path that triggered abuse classification (telltale
+  path or the 404 that crossed the threshold),
+- `is_404` — true for 404 responses, false for document GETs from the
+  abuser.
+
+Crawler hits are grouped by client hash in the analytics viewer; abuse hits
+are grouped by IP alone (resolved from the referenced `Client`).  In the
+Abuse table identical paths are collapsed with their counts; flagged paths
+that triggered classification are lifted to the top, followed by other 404s
+and then document GETs from the abuser.  Within each category paths are
+sorted by count descending, then by their earliest hit.
 
 ## Aggregates
 
@@ -148,8 +196,8 @@ with a "could not be loaded" message.
 Because it is a real page, fetch-navigation handles it like any other internal
 link: clicking the 📊 pen (or any link to `/_a`) fetches the server-rendered
 HTML, swaps the dynamic regions and mounts the Vue analytics app in place. The
-range selector updates the URL query string (`?range=week` etc.) so links to
-a specific range can be shared.
+range selector updates the URL hash (`#week` etc.) so links to a specific
+range can be shared.
 
 `AnalyticsView.vue` is no longer a full-screen overlay; the `body.analytics-open`
 page-chrome hiding and `#/analytics/<range>` hash routing have been removed.
@@ -188,9 +236,13 @@ carrying less than 1% of the total traffic
 pruned; beads are simulated one by one in JS (requestAnimationFrame) and
 flow along each edge, emitted at a rate linearly proportional
 to the directional count with no in-flight limit, opposing directions
-offset onto parallel lanes. External referers show as a node row above the
-map, external exits as small nodes fanned outwards from their source
-page), per-page view
+offset onto parallel lanes. External sources show as a node row above the
+map: each visit is attributed to `utm_campaign`, then `utm_source`, then the
+referer origin, then any other `utm_*` tag, so UTM-tagged visits are grouped
+under their campaign/source value rather than the referer domain. A UTM
+source node only links to its referer when every visit carrying that tag
+came from the same origin. External exits are small nodes fanned outwards
+from their source page), per-page view
 counts, the top transitions and the 50 most recent visit trails. Data is
 streamed live over `WebSocket /_api/ws/analytics`, which pushes the latest
 JSON snapshot on connect and again whenever the analytics file is updated

frontend/src/AnalyticsView.vue

@@ -7,12 +7,17 @@
 import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
 import { RANGES } from './analytics/time.js'
 import {
+  calcReadStats,
   calcTotalViews,
   copyIp,
+  copyList,
+  formatCount,
+  formatAbuseRows,
   formatCrawlerRows,
   formatVisitRows,
 } from './analytics/format.js'
-import * as flagSvgs from 'country-flag-icons/string/3x2'
+import TrailLink from './TrailLink.vue'
+import VisitorCell from './VisitorCell.vue'
 import TransitionGraph from './TransitionGraph.vue'
 import VisitorCharts from './VisitorCharts.vue'
 
@@ -20,6 +25,8 @@ const props = defineProps({
   initialRange: { type: String, default: 'week' },
 })
 
+const ABUSE_MAX_LINES = 5
+
 const data = ref(null)
 const pageTree = ref(null)
 const error = ref('')
@@ -52,7 +59,7 @@ function connectAnalytics() {
 onMounted(async () => {
   connectAnalytics()
   now.value = Date.now()
-  timeInterval = setInterval(() => { now.value = Date.now() }, 30000)
+  timeInterval = setInterval(() => { now.value = Date.now() }, 1000)
   // The site tree for the transition map (all pages in menu order). Not
   // fatal: without it the map just narrows to pages seen in transitions.
   try {
@@ -73,32 +80,23 @@ onUnmounted(() => {
 
 const visits = computed(() => data.value?.visits || [])
 const totalViews = computed(() => calcTotalViews(data.value?.views))
+const readStats = computed(() => calcReadStats(visits.value))
 
 const range = ref(RANGES[props.initialRange] ? props.initialRange : 'week')
 
 // Keep the URL shareable when the range changes.
 watch(range, (r) => {
   const url = new URL(location.href)
-  url.searchParams.set('range', r)
+  url.hash = r
   history.replaceState(null, '', url)
 })
 
-const visitRows = computed(() => formatVisitRows(visits.value, pageTree.value, now.value))
+const clients = computed(() => data.value?.clients || {})
+const visitRows = computed(() => formatVisitRows(visits.value, clients.value, pageTree.value, now.value))
 const crawlers = computed(() => data.value?.crawlers || [])
-const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, pageTree.value, now.value))
+const crawlerRows = computed(() => formatCrawlerRows(crawlers.value, clients.value, pageTree.value, now.value))
+const abuseRows = computed(() => formatAbuseRows(data.value?.abuse || [], clients.value, now.value))
 
-function flagSvg(code) {
-  return flagSvgs[code?.toUpperCase()] || ''
-}
-
-function countryName(code) {
-  if (!code) return ''
-  try {
-    return new Intl.DisplayNames(['en'], { type: 'region' }).of(code.toUpperCase())
-  } catch {
-    return ''
-  }
-}
 </script>
 
 <template>
@@ -118,8 +116,10 @@ function countryName(code) {
       <p v-else-if="!data" class="loading">loading…</p>
       <template v-else>
         <section class="totals">
-          <div><strong>{{ visits.length }}</strong> visits</div>
-          <div><strong>{{ totalViews }}</strong> page views</div>
+          <div><strong :title="String(visits.length)">{{ formatCount(visits.length) }}</strong> visits</div>
+          <div><strong :title="String(totalViews)">{{ formatCount(totalViews) }}</strong> page views</div>
+          <div><strong>{{ readStats.avgMinPerVisit }}</strong> min/visit</div>
+          <div><strong>{{ readStats.avgArticleMedianMin }}</strong> min article read</div>
         </section>
 
         <VisitorCharts :data="data" :range="range" />
@@ -131,83 +131,112 @@ function countryName(code) {
             <table class="visit-table">
               <thead>
                 <tr>
-                  <th>when</th>
                   <th>trail</th>
-                  <th>referer</th>
-                  <th>ip</th>
-                  <th>lang</th>
-                  <th>country</th>
-                  <th>ua</th>
-                  <th>utm</th>
+                  <th>visitor</th>
+                  <th class="last-seen">last seen</th>
                 </tr>
               </thead>
               <tbody>
                 <tr v-for="(v, i) in visitRows" :key="i">
-                  <td class="when" :title="v.whenTooltip">{{ v.when }}</td>
                   <td class="trail">
-                    <a v-for="(s, si) in v.trail" :key="si"
-                       :href="s.path" :title="s.title"
-                       :target="s.external ? '_blank' : undefined"
-                       :rel="s.external ? 'noopener' : undefined"
-                       @click="(e) => { if (!s.external) $emit('close') }">
-                      {{ s.slug }}
-                    </a>
+                    <TrailLink v-if="v.refererStep" :step="v.refererStep" @close="$emit('close')" />
+                    <span v-if="v.utm && v.utm !== '—'" class="utm-tag small muted" :title="v.utmTitle">{{ v.utm }}</span>
+                    <TrailLink v-for="(s, si) in v.trail" :key="si" :step="s" @close="$emit('close')" />
                   </td>
-                  <td>{{ v.referer }}</td>
-                  <td>
-                    <span class="clickable-ip"
-                          :title="`Click to copy full IP: ${v.ip}`"
-                          @click="copyIp(v.ip)">{{ v.ipDisplay }}</span>
-                  </td>
-                  <td>{{ v.lang }}</td>
-                  <td class="country">
-                    <span v-if="flagSvg(v.country)" class="flag" v-html="flagSvg(v.country)" :title="countryName(v.country) || v.country"></span>
-                    <template v-if="v.city !== '—'">{{ countryName(v.country) || v.country }}<br><small class="muted">{{ v.city }}</small></template>
-                    <template v-else-if="v.country !== '—'">{{ countryName(v.country) || v.country }}</template>
-                    <template v-else>—</template>
-                  </td>
-                  <td class="ua" :title="v.uaRaw">{{ v.ua }}</td>
-                  <td>{{ v.utm }}</td>
+                  <VisitorCell
+                    :ip="v.ip"
+                    :ip-display="v.ipDisplay"
+                    :ua="v.ua"
+                    :ua-raw="v.uaRaw"
+                    :country="v.country"
+                    :city="v.city"
+                    :lang="v.lang"
+                    :lang-display="v.langDisplay"
+                    :is-host="v.isHost"
+                  />
+                  <td class="last-seen muted"
+                      :title="v.lastSeenLocal"
+                      @click="copyList(v.lastSeenIso, $event)">{{ v.lastSeen }}</td>
                 </tr>
               </tbody>
             </table>
           </div>
           <p v-else class="empty">no visits recorded yet</p>
-        </section>
 
-        <section>
-          <h2>Crawlers</h2>
           <div v-if="crawlerRows.length" class="visit-table-wrap">
             <table class="visit-table">
               <thead>
                 <tr>
-                  <th>when</th>
-                  <th>pages</th>
-                  <th>ip</th>
-                  <th>ua</th>
+                  <th>pages crawled</th>
+                  <th>visitor</th>
+                  <th class="last-seen">last seen</th>
                 </tr>
               </thead>
               <tbody>
                 <tr v-for="(c, i) in crawlerRows" :key="i">
-                  <td class="when" :title="c.whenTooltip">{{ c.when }}</td>
                   <td class="trail">
-                    <a v-for="(s, si) in c.pages" :key="si"
-                       :href="s.path" :title="`${s.title}${s.count > 1 ? ` (${s.count} hits)` : ''}`"
-                       @click="$emit('close')">
-                      <small v-if="s.count > 1" class="muted">{{ s.count }}×</small>{{ s.slug }}
-                    </a>
+                    <TrailLink v-for="(s, si) in c.pages" :key="si" :step="s" :count="s.count" @close="$emit('close')" />
                   </td>
-                  <td>
-                    <span class="clickable-ip"
-                          :title="`Click to copy full IP: ${c.ip}`"
-                          @click="copyIp(c.ip)">{{ c.ipDisplay }}</span>
-                  </td>
-                  <td class="ua" :title="c.uaRaw">{{ c.ua }}</td>
+                  <VisitorCell
+                    :ip="c.ip"
+                    :ip-display="c.ipDisplay"
+                    :ua="c.ua"
+                    :ua-raw="c.uaRaw"
+                    :country="c.country"
+                    :city="c.city"
+                    :lang="c.lang"
+                    :lang-display="c.langDisplay"
+                    :is-host="c.isHost"
+                  />
+                  <td class="last-seen muted"
+                      :title="c.lastSeenLocal"
+                      @click="copyList(c.lastSeenIso, $event)">{{ c.lastSeen }}</td>
                 </tr>
               </tbody>
             </table>
           </div>
           <p v-else class="empty">no crawler hits recorded yet</p>
+
+          <div v-if="abuseRows.length" class="visit-table-wrap">
+            <table class="visit-table">
+              <thead>
+                <tr>
+                  <th>paths abused</th>
+                  <th>visitor</th>
+                  <th class="last-seen">last seen</th>
+                </tr>
+              </thead>
+              <tbody>
+                <tr v-for="(a, i) in abuseRows" :key="i">
+                  <td class="trail abuse-list clickable-list"
+                      @click="copyList(a.allPaths, $event)">
+                    <div class="abuse-items">
+                      <span v-for="(p, pi) in a.paths.slice(0, ABUSE_MAX_LINES)" :key="pi"
+                            class="inline-item">
+                        <small v-if="p.count > 1" class="muted">{{ formatCount(p.count) }}×</small>{{ p.path }}
+                      </span>
+                      <small v-if="a.paths.length > ABUSE_MAX_LINES" class="muted">+{{ a.paths.length - ABUSE_MAX_LINES }} more</small>
+                    </div>
+                  </td>
+                  <VisitorCell
+                    :ip="a.ip"
+                    :ip-display="a.ipDisplay"
+                    :ua="a.ua"
+                    :ua-raw="a.uaRaw"
+                    :country="a.country"
+                    :city="a.city"
+                    :lang="a.lang"
+                    :lang-display="a.langDisplay"
+                    :is-host="a.isHost"
+                    :variant-count="a.clientCount"
+                  />
+                  <td class="last-seen muted"
+                      :title="a.lastSeenLocal"
+                      @click="copyList(a.lastSeenIso, $event)">{{ a.lastSeen }}</td>
+                </tr>
+              </tbody>
+            </table>
+          </div>
         </section>
       </template>
     </div>
@@ -282,6 +311,15 @@ function countryName(code) {
   margin-top: 1.8rem;
 }
 
+.analytics-view a {
+  color: var(--text);
+  text-decoration: none;
+}
+.analytics-view a:hover { color: var(--accent); }
+
+.analytics-view :deep(.muted) { color: var(--muted); }
+.analytics-view :deep(.small) { font-size: 0.75em; }
+
 .totals {
   display: flex;
   gap: 2rem;
@@ -296,7 +334,6 @@ function countryName(code) {
 .visit-table {
   width: 100%;
   border-collapse: collapse;
-  font-family: monospace;
   font-size: 0.82rem;
   line-height: 1.3;
 }
@@ -318,9 +355,11 @@ function countryName(code) {
   background: var(--bg, Canvas);
 }
 
-.visit-table .when {
+.visit-table .last-seen {
+  width: 5rem;
+  text-align: right;
   white-space: nowrap;
-  color: var(--muted);
+  cursor: pointer;
 }
 
 .visit-table .trail {
@@ -328,54 +367,74 @@ function countryName(code) {
   overflow-wrap: break-word;
 }
 
-.visit-table .trail a {
-  color: var(--text);
-  text-decoration: none;
+.visit-table .trail a,
+.visit-table .trail-link {
+  display: inline-block;
+  max-width: 8rem;
+  white-space: nowrap;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  vertical-align: bottom;
 }
 
-.visit-table .trail a:hover { color: var(--accent); }
-
-.visit-table .trail a + a {
+.visit-table .trail > * + * {
   margin-left: 0.5rem;
 }
 
-.visit-table .trail small,
-.visit-table small.muted {
-  color: var(--muted);
-  font-size: 0.75em;
-}
-
-.visit-table .clickable-ip {
-  cursor: pointer;
-  text-decoration: underline;
-  text-decoration-style: dotted;
-}
-
-.visit-table .clickable-ip:hover {
-  color: var(--accent);
-}
-
-.visit-table .ua {
-  max-width: 18rem;
+.visit-table .utm-tag {
+  display: inline-block;
+  max-width: 100%;
+  padding: 0.05rem 0.4rem;
+  border: 1px solid var(--line);
+  border-radius: 0.25rem;
+  white-space: nowrap;
   overflow: hidden;
   text-overflow: ellipsis;
+  vertical-align: bottom;
+}
+
+.visit-table .clickable-list {
+  cursor: pointer;
+  max-width: 22rem;
+}
+
+.visit-table .abuse-items {
+  display: flex;
+  flex-wrap: wrap;
+  gap: 0.15rem 0.5rem;
+  align-items: baseline;
+}
+
+.visit-table .inline-item {
+  max-width: 18rem;
+  min-width: 0;
   white-space: nowrap;
-}
-
-.visit-table .country .flag {
-  display: inline-flex;
-  width: 18px;
-  height: 12px;
-  border-radius: 2px;
   overflow: hidden;
-  border: 1px solid var(--line);
-  box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
+  text-overflow: ellipsis;
+  word-break: keep-all;
+  hyphens: none;
 }
 
-.visit-table .country .flag :deep(svg) {
-  width: 100%;
-  height: 100%;
-  display: block;
+.visit-table :deep(.clickable-ip),
+.visit-table .clickable-list,
+.visit-table .last-seen {
+  cursor: pointer;
+  position: relative;
+}
+
+.visit-table :deep(.copy-popup) {
+  position: absolute;
+  bottom: calc(100% + 0.25rem);
+  left: 50%;
+  transform: translateX(-50%);
+  padding: 0.15rem 0.4rem;
+  background: var(--text, CanvasText);
+  color: var(--bg, Canvas);
+  border-radius: 0.25rem;
+  font-size: 0.75rem;
+  white-space: nowrap;
+  pointer-events: none;
+  z-index: 10;
 }
 
 .crawler-top-uas {

frontend/src/TrailLink.vue

@@ -0,0 +1,22 @@
+<script setup>
+import { formatCount } from './analytics/format.js'
+
+defineProps({
+  step: { type: Object, required: true },
+  count: { type: Number, default: 0 },
+})
+
+defineEmits(['close'])
+</script>
+
+<template>
+  <a class="trail-link"
+     :href="step.path"
+     :title="count > 1 ? `${step.title} (${count} hits)` : step.title"
+     :target="step.external ? '_blank' : undefined"
+     :rel="step.external ? 'noopener' : undefined"
+     @click="(e) => { if (!step.external) $emit('close') }">
+    <small v-if="count > 1" class="muted">{{ formatCount(count) }}×</small>
+    <span>{{ step.slug }}</span>
+  </a>
+</template>

frontend/src/TransitionGraph.vue

@@ -7,7 +7,8 @@
  * range, exactly like the charts and per-page views do.
  */
 import { computed, onBeforeUnmount, shallowRef, watch } from 'vue'
-import { rangeWindow } from './analytics/time.js'
+import { rangeWindow, WEEK } from './analytics/time.js'
+import { formatCount } from './analytics/format.js'
 import {
   TNODE_R,
   BEAD_R,
@@ -15,6 +16,7 @@ import {
   buildTransitionGraph,
   filterTransitionsByRange,
   filterViewsByRange,
+  filterVisitsByRange,
 } from './analytics/transitions.js'
 
 const props = defineProps({
@@ -25,6 +27,19 @@ const props = defineProps({
 
 const window = computed(() => rangeWindow(props.range))
 
+const visualScale = computed(() => {
+  const { t0, t1 } = window.value
+  if (t0 != null && t1 != null) return WEEK / (t1 - t0)
+  // 'all': scale by the actual data span.
+  const times = new Set()
+  for (const buckets of Object.values(props.data?.views || {})) {
+    for (const k of Object.keys(buckets)) times.add(Date.parse(k))
+  }
+  const arr = [...times]
+  if (arr.length < 2) return 1
+  return WEEK / (Math.max(...arr) - Math.min(...arr))
+})
+
 const filteredData = computed(() => {
   if (!props.data) return null
   const { t0, t1 } = window.value
@@ -34,9 +49,13 @@ const filteredData = computed(() => {
   }
 })
 
+const filteredVisits = computed(() =>
+  filterVisitsByRange(props.data?.visits, window.value.t0, window.value.t1),
+)
+
 const graph = computed(() =>
   filteredData.value
-    ? buildTransitionGraph(filteredData.value, props.pageTree)
+    ? buildTransitionGraph(filteredData.value, props.pageTree, filteredVisits.value, visualScale.value)
     : null,
 )
 
@@ -47,16 +66,24 @@ const graph = computed(() =>
 const beads = shallowRef([])
 let rafId = 0
 
+const MAX_BEAD_RATE = 120 // upper bound on total beads per second
+
 const startBeads = (flows) => {
   cancelAnimationFrame(rafId)
   beads.value = []
   if (!flows?.length) return
   if (matchMedia('(prefers-reduced-motion: reduce)').matches) return
 
+  // Cap the total bead emission rate so a busy range cannot spawn enough
+  // beads to kill the page. Existing per-range time scaling is preserved;
+  // this is only a proportional emergency throttle when the limit is hit.
+  const totalRate = flows.reduce((s, f) => s + 1 / f.interval, 0)
+  const scale = totalRate > MAX_BEAD_RATE ? MAX_BEAD_RATE / totalRate : 1
+
   const live = [] // { flow, t0 } — one entry per bead in flight
   const now = performance.now()
   const emitters = flows.map((flow) => {
-    const interval = flow.interval * 1000
+    const interval = (flow.interval / scale) * 1000
     // Pre-fill the traversal with evenly spaced beads (random phase), so
     // the flow appears already running instead of starting empty.
     const phase = Math.random() * interval
@@ -100,28 +127,56 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
   <section v-if="graph">
     <svg class="tmap" :viewBox="`${graph.bounds.x0} ${graph.bounds.y0} ${graph.bounds.x1 - graph.bounds.x0} ${graph.bounds.y1 - graph.bounds.y0}`"
          role="img" aria-label="map of transitions between pages">
+      <defs>
+        <!-- Unit-radius circle; only the portion near the bottom is used. -->
+        <path id="tnode-label-arc" d="M 0,-1 A 1,1 0 1,0 0,1 A 1,1 0 1,0 -0.001,-1" />
+      </defs>
       <path v-for="(a, i) in graph.arcs" :key="'a' + i"
             :d="a.d" class="tarc" />
       <path v-for="(e, i) in graph.edges" :key="'e' + i"
-            :d="e.d" class="tconn">
+            :d="e.d" :class="['tconn', e.external && 'tconn-exit']">
         <title>{{ e.title }}</title>
       </path>
       <circle v-for="(b, i) in beads" :key="'b' + i"
               :cx="b.x" :cy="b.y" :r="BEAD_R" class="tbead" />
       <g v-for="(x, i) in graph.extNodes" :key="'x' + i">
-        <a :href="x.path" target="_blank" rel="noopener" :title="x.path">
+        <a v-if="x.href" :href="x.href" target="_blank" rel="noopener" :title="x.path">
           <circle :cx="x.x" :cy="x.y" :r="x.r"
                   :class="['txnode', x.kind === 'source' ? 'txnode-source' : 'txnode-exit']" />
-          <text :x="x.x" :y="x.y - 2" class="tnodeslug">{{ x.label }}</text>
-          <text :x="x.x" :y="x.y + 12" class="tnodecount">{{ x.count }}</text>
+          <text :transform="`translate(${x.x}, ${x.y}) scale(${x.r - 4})`" class="tnodeslug" :style="{ '--node-r': x.r - 4 }">
+            <textPath href="#tnode-label-arc" startOffset="50%" text-anchor="middle" side="right">{{ x.label }}</textPath>
+          </text>
+          <text :x="x.x" :y="x.y + 4" class="tnodecount">{{ formatCount(x.count) }}</text>
         </a>
+        <g v-else :title="x.path">
+          <circle :cx="x.x" :cy="x.y" :r="x.r"
+                  :class="['txnode', x.kind === 'source' ? 'txnode-source' : 'txnode-exit']" />
+          <text :transform="`translate(${x.x}, ${x.y}) scale(${x.r - 4})`" class="tnodeslug" :style="{ '--node-r': x.r - 4 }">
+            <textPath href="#tnode-label-arc" startOffset="50%" text-anchor="middle" side="right">{{ x.label }}</textPath>
+          </text>
+          <text :x="x.x" :y="x.y + 4" class="tnodecount">{{ formatCount(x.count) }}</text>
+        </g>
       </g>
       <g v-for="n in graph.nodes" :key="n.path">
-        <a :href="n.path" :title="n.title">
+        <a v-if="!n.hidden" :href="n.path" :title="n.title">
           <circle :cx="n.x" :cy="n.y" :r="TNODE_R" class="tnode" />
-          <text :x="n.x" :y="n.y - 2" class="tnodeslug">{{ n.label }}</text>
-          <text :x="n.x" :y="n.y + 12" class="tnodecount">{{ n.views }}</text>
+          <text :transform="`translate(${n.x}, ${n.y}) scale(${TNODE_R - 4})`" class="tnodeslug" :style="{ '--node-r': TNODE_R - 4 }">
+            <textPath href="#tnode-label-arc" startOffset="50%" text-anchor="middle" side="right">{{ n.label }}</textPath>
+          </text>
+          <text :x="n.x" :y="n.y + 4" class="tnodecount">
+            {{ n.readMin ? `${formatCount(n.views)}×${n.readMin}m` : formatCount(n.views) }}
+          </text>
         </a>
+        <template v-else>
+          <text :x="n.x" :y="n.y"
+                :transform="`rotate(${n.angle * 180 / Math.PI}, ${n.x}, ${n.y})`"
+                class="tnodehidden" text-anchor="start" dominant-baseline="middle">➤</text>
+          <text :x="n.x + Math.cos(n.angle) * 10"
+                :y="n.y + Math.sin(n.angle) * 10"
+                :transform="`rotate(${(n.angle + (Math.cos(n.angle) < 0 ? Math.PI : 0)) * 180 / Math.PI}, ${n.x + Math.cos(n.angle) * 10}, ${n.y + Math.sin(n.angle) * 10})`"
+                :text-anchor="Math.cos(n.angle) < 0 ? 'end' : 'start'"
+                class="tnodehidden" dominant-baseline="middle">{{ n.label }}</text>
+        </template>
       </g>
     </svg>
   </section>
@@ -139,6 +194,9 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
   fill: var(--accent);
   opacity: 0.4; /* uniform, not strength-encoded: width carries that */
 }
+.tmap .tconn-exit {
+  fill: var(--text);
+}
 .tmap .tbead {
   fill: var(--accent);
   opacity: 0.85;
@@ -149,7 +207,7 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
   stroke-width: 1.5;
 }
 .tmap .txnode-source { stroke: var(--text); }
-.tmap .txnode-exit { stroke: var(--muted); }
+.tmap .txnode-exit { stroke: var(--text); }
 .tmap .tarc {
   fill: none;
   stroke: var(--line);
@@ -162,7 +220,7 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
 }
 .tmap .tnodeslug {
   fill: var(--text);
-  font-size: 11px;
+  font-size: calc(11px / var(--node-r, 34));
   text-anchor: middle;
 }
 .tmap a { cursor: pointer; }
@@ -172,6 +230,10 @@ onBeforeUnmount(() => cancelAnimationFrame(rafId))
   font-size: 10px;
   text-anchor: middle;
 }
+.tmap .tnodehidden {
+  fill: var(--text);
+  font-size: 9px;
+}
 
 section { margin-top: 1.8rem; }
 </style>

frontend/src/VisitorCell.vue

@@ -0,0 +1,156 @@
+<script setup>
+// Visitor metadata cell shared by the recent-visits, crawlers, and abuse tables.
+// Displays IP/network/host, country flag/city, UA, and language when available.
+// Clicking the IP copies the full address to the clipboard.
+// ``variantCount`` overrides the UA line to warn when multiple client
+// fingerprints share the same IP (e.g. a scanner rotating UAs).
+import { computed } from 'vue'
+import * as flagSvgs from 'country-flag-icons/string/3x2'
+import { copyIp, formatLang } from './analytics/format.js'
+
+const props = defineProps({
+  ip: { type: String, default: '' },
+  ipDisplay: { type: String, default: '—' },
+  ua: { type: String, default: '' },
+  uaRaw: { type: String, default: '' },
+  country: { type: String, default: '' },
+  city: { type: String, default: '' },
+  lang: { type: String, default: '' },
+  langDisplay: { type: String, default: '' },
+  isHost: { type: Boolean, default: false },
+  variantCount: { type: Number, default: 1 },
+})
+
+const hasCountry = computed(() => !!(props.country && props.country !== '—'))
+const hasCity = computed(() => !!(props.city && props.city !== '—'))
+const hasLocale = computed(() => hasCountry.value || hasCity.value)
+const langValue = computed(() => props.langDisplay || formatLang(props.lang))
+const showLang = computed(() => langValue.value && langValue.value !== '—')
+
+function flagSvg(code) {
+  return flagSvgs[code?.toUpperCase()] || ''
+}
+
+function countryName(code) {
+  if (!code) return ''
+  try {
+    return new Intl.DisplayNames(['en'], { type: 'region' }).of(code.toUpperCase())
+  } catch {
+    return ''
+  }
+}
+</script>
+
+<template>
+  <td class="visitor-cell" :class="{ 'host-cell': isHost }">
+    <div class="visitor-rows">
+      <div class="visitor-row">
+        <div class="locale-line">
+          <span v-if="flagSvg(country)" class="flag" v-html="flagSvg(country)" :title="countryName(country) || country"></span>
+          <template v-if="hasCity"><small class="city-name muted">{{ city }}</small></template>
+          <template v-else-if="!hasLocale">—</template>
+        </div>
+        <div class="ip-line">
+          <span class="clickable-ip small muted"
+                :title="ip"
+                @click="copyIp(ip, $event)">{{ ipDisplay }}</span>
+        </div>
+      </div>
+      <div class="visitor-row">
+        <div class="ua-line">
+          <small v-if="variantCount > 1" class="muted variant-hint">{{ variantCount }} client variations</small>
+          <small v-else class="muted" :title="uaRaw">{{ ua || '—' }}</small>
+        </div>
+        <div v-if="showLang && variantCount <= 1" class="locale-lang"><small class="muted">{{ langValue }}</small></div>
+      </div>
+    </div>
+  </td>
+</template>
+
+<style scoped>
+.visitor-cell {
+  width: 18em;
+  max-width: 18em;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  vertical-align: top;
+}
+
+.visitor-cell.host-cell {
+  text-align: right;
+}
+
+.visitor-rows {
+  display: flex;
+  flex-direction: column;
+  gap: 0.15rem;
+}
+
+.visitor-row {
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  gap: 0.5rem;
+}
+
+.visitor-row > * {
+  min-width: 0;
+}
+
+.locale-line,
+.ip-line,
+.ua-line {
+  flex: 1 1 auto;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  white-space: nowrap;
+}
+
+.locale-line {
+  text-align: left;
+  display: flex;
+  align-items: center;
+  gap: 0.3rem;
+}
+
+.ip-line {
+  text-align: right;
+}
+
+.ua-line {
+  text-align: left;
+}
+
+.locale-lang {
+  flex: 0 0 auto;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  white-space: nowrap;
+  text-align: right;
+}
+
+.city-name {
+  display: inline-block;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  white-space: nowrap;
+  vertical-align: middle;
+}
+
+.flag {
+  display: inline-flex;
+  width: 18px;
+  height: 12px;
+  border-radius: 2px;
+  overflow: hidden;
+  border: 1px solid var(--line);
+  box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.2) inset;
+  vertical-align: middle;
+}
+
+.flag :deep(svg) {
+  width: 100%;
+  height: 100%;
+  display: block;
+}
+</style>

frontend/src/VisitorCharts.vue

@@ -2,10 +2,12 @@
 /**
  * Visitor and page-view smoothed curves for a single shared time range.
  */
-import { computed } from 'vue'
+import { computed, onMounted, onUnmounted, ref } from 'vue'
 import { makeSeries } from './analytics/time.js'
 import { CHART_H, CHART_W, buildChart, fmtY } from './analytics/chart.js'
 
+const DAY_REFRESH_MS = 15000
+
 const props = defineProps({
   data: { type: Object, default: null },
   range: { type: String, required: true },
@@ -22,33 +24,52 @@ const allViews = computed(() => {
 
 const visitSeries = computed(() => makeSeries(props.data?.site_visits, props.range))
 const viewSeries = computed(() => makeSeries(allViews.value, props.range))
-const unit = computed(() => (props.range === 'week' ? 'h' : 'day'))
 
-const visitChart = computed(() => buildChart(visitSeries.value))
-const viewChart = computed(() => buildChart(viewSeries.value))
+function freqLabel(unit) {
+  return unit === '5min' ? '5 min' : unit === 'hour' ? 'hourly' : 'daily'
+}
+
+const now = ref(Date.now())
+let refreshInterval = null
+onMounted(() => {
+  refreshInterval = setInterval(() => { now.value = Date.now() }, DAY_REFRESH_MS)
+})
+onUnmounted(() => {
+  if (refreshInterval) clearInterval(refreshInterval)
+})
+
+const visitChart = computed(() => buildChart(visitSeries.value, now.value))
+const viewChart = computed(() => buildChart(viewSeries.value, now.value))
 </script>
 
 <template>
   <section v-for="c in [
-      { ylabel: 'visitors', chart: visitChart, empty: 'no visits recorded yet' },
+      { ylabel: 'visits', chart: visitChart, empty: 'no visits recorded yet' },
       { ylabel: 'views', chart: viewChart, empty: 'no views recorded yet' },
     ]" :key="c.ylabel">
     <template v-if="c.chart">
       <div class="chartwrap">
         <div class="plot">
           <div class="plotarea">
-            <span class="yaxis-label">{{ c.ylabel }}/{{ unit }}</span>
+            <span class="yaxis-label">{{ freqLabel(c.chart.unit) }} {{ c.ylabel }}</span>
             <svg class="chart" :viewBox="`0 0 ${CHART_W} ${CHART_H}`"
-                 preserveAspectRatio="none" role="img" :aria-label="`${c.ylabel} per ${unit}`">
+                 preserveAspectRatio="none" role="img" :aria-label="`${freqLabel(c.chart.unit)} ${c.ylabel}`">
               <line v-for="g in c.chart.majors.slice(1)" :key="'j' + g.value"
                     :x1="0" :x2="CHART_W" :y1="g.y" :y2="g.y" class="major" />
               <template v-for="t in c.chart.xticks" :key="'t' + t.x">
                 <line v-if="t.line" :x1="t.x" :x2="t.x" :y1="0" :y2="CHART_H"
                       class="minor vertical" />
               </template>
-              <template v-for="(s, i) in c.chart.series" :key="i">
-                <path v-if="s.area" :d="s.area" class="area" />
-                <path :d="s.line" class="line" :style="{ opacity: s.opacity }" />
+              <template v-if="c.chart.bars">
+                <rect v-for="(b, i) in c.chart.bars" :key="'b' + i"
+                      :x="b.x" :y="b.y" :width="b.width" :height="b.height" class="bar" />
+                <path :d="c.chart.skyline" class="line" />
+              </template>
+              <template v-else>
+                <template v-for="(s, i) in c.chart.series" :key="i">
+                  <path v-if="s.area" :d="s.area" class="area" />
+                  <path :d="s.line" class="line" :style="{ opacity: s.opacity }" />
+                </template>
               </template>
               <line :x1="0" :x2="CHART_W" :y1="CHART_H - 0.5" :y2="CHART_H - 0.5"
                     class="axis" />
@@ -62,7 +83,7 @@ const viewChart = computed(() => buildChart(viewSeries.value))
           </div>
         </div>
       </div>
-      <div v-if="c.chart.series.length > 1" class="legend">
+      <div v-if="c.chart.series && c.chart.series.length > 1" class="legend">
         <span v-for="(s, i) in c.chart.series" :key="i" :style="{ opacity: s.opacity }">
           ● {{ s.label }}
         </span>
@@ -76,7 +97,7 @@ const viewChart = computed(() => buildChart(viewSeries.value))
 /* The svg is stretched (preserveAspectRatio none), so all text lives in
    HTML overlays positioned by the same fractions the geometry uses. */
 .chartwrap {
-  padding-left: 2.2rem; /* y labels */
+  padding-left: 2.8rem; /* y labels */
 }
 
 .plot {
@@ -103,8 +124,8 @@ const viewChart = computed(() => buildChart(viewSeries.value))
 
 .ylab {
   position: absolute;
-  left: -2.2rem;
-  width: 1.9rem;
+  left: -2.8rem;
+  width: 2.6rem;
   text-align: right;
   transform: translateY(50%);
   font-size: 0.7rem;
@@ -154,6 +175,11 @@ const viewChart = computed(() => buildChart(viewSeries.value))
   opacity: 0.15;
 }
 
+.chart .bar {
+  fill: var(--accent);
+  opacity: 0.15;
+}
+
 .chart .line {
   fill: none;
   stroke: var(--accent);
@@ -176,10 +202,11 @@ const viewChart = computed(() => buildChart(viewSeries.value))
 .yaxis-label {
   position: absolute;
   top: 50%;
-  left: -2.2rem;
+  left: -2.8rem;
   font-size: 0.7rem;
   color: var(--muted);
   writing-mode: vertical-rl;
+  white-space: nowrap;
   transform: translateY(-50%) rotate(180deg);
 }
 

frontend/src/analytics-main.js

@@ -9,7 +9,7 @@ let app = null
 export function mount(container) {
   if (app) return
   app = createApp(AnalyticsView, {
-    initialRange: new URLSearchParams(location.search).get('range') || 'week',
+    initialRange: location.hash.slice(1) || 'week',
   })
   app.mount(container)
 }

frontend/src/analytics/chart.js

@@ -5,7 +5,8 @@
  * rates (hour on the week view, day on month+).
  */
 
-import { DAY, HOUR, WEEK, mondayUTC } from './time.js'
+import { DAY, HOUR, MIN5, WEEK, mondayUTC } from './time.js'
+import { formatCount } from './format.js'
 
 export const CHART_W = 720
 export const CHART_H = 180
@@ -206,9 +207,10 @@ export function spline(pts) {
 }
 
 /** Build a full chart model from a series descriptor produced by time.js. */
-export function buildChart(input) {
+export function buildChart(input, now = Date.now()) {
   if (!input || !input.series.length) return null
-  const { series, t0, t1, rate, binMinutes, unitMinutes } = input
+  if (input.unit === '5min') return buildDayChart(input, now)
+  const { series, t0, t1, rate, binMinutes, unitMinutes, unit } = input
   // Values are per-unit rates (hour on the week view, day on month+); the
   // y max is derived from the *smoothed* curves so random single-bucket
   // spikes don't blow up the scale. Smoothing works on raw counts (its edge
@@ -283,7 +285,91 @@ export function buildChart(input) {
           x: x(t), left: ((t - t0) / (t1 - t0)) * 100,
           label: fmtTick(t, t1 - t0), line: true,
         }))
-  return { max, majors, minors, series: drawn, xticks }
+  return { max, majors, minors, series: drawn, xticks, unit }
+}
+
+/**
+ * Day view: 5-minute bars for the last 24 hours. Bars are drawn at raw
+ * counts; the skyline uses a projected full-bucket value for the still-open
+ * final bucket. The y scale is derived from the projected skyline maximum.
+ */
+export function buildDayChart(input, now = Date.now()) {
+  const { series, t0, t1 } = input
+  const points = series[0]?.points || []
+  const n = points.length
+  if (!n) return null
+  const bucketMs = (t1 - t0) / n
+  const bucketWidth = CHART_W / n
+  const gap = 0.2
+  const barWidth = Math.max(0.2, bucketWidth - gap)
+
+  const x = (i) => i * bucketWidth + gap / 2
+  const prevRaw = n > 1 ? points[n - 2].count : 0
+  const projected = points.map((p, i) => {
+    if (i !== n - 1) return p.count
+    const bucketStart = t0 + i * bucketMs
+    const elapsed = Math.max(1, Math.min(bucketMs, now - bucketStart))
+    // Blend the observed partial bucket with the previous full bucket:
+    // the longer the current bucket has run, the less we borrow from it.
+    const share = elapsed / bucketMs
+    return p.count + prevRaw * (1 - share)
+  })
+  const highest = Math.max(0, ...projected)
+  const { max, step, minor } = yScale(highest)
+  const y = (v) => PAD_TOP + (1 - Math.max(0, v) / max) * (CHART_H - PAD_TOP)
+
+  const bars = points.map((p, i) => {
+    const bx = x(i)
+    const by = y(p.count)
+    return {
+      x: bx,
+      y: by,
+      width: barWidth,
+      height: CHART_H - by,
+      raw: p.count,
+      projected: projected[i],
+    }
+  })
+
+  let skyline = ''
+  for (let i = 0; i < bars.length; i++) {
+    const b = bars[i]
+    const top = y(b.projected)
+    if (i === 0) {
+      skyline += `M${b.x},${top} H${b.x + b.width}`
+    } else {
+      skyline += ` V${top} H${b.x + b.width}`
+    }
+  }
+
+  const majors = []
+  const minors = []
+  const nMajor = Math.round(max / step)
+  for (let k = 0; k <= nMajor; k++) {
+    const v = k * step
+    majors.push({ value: v, y: y(v), bottom: (1 - PAD_TOP / CHART_H) * (v / max) * 100 })
+  }
+  if (minor) {
+    for (let v = minor; v < max; v += minor) {
+      if (v % step !== 0) minors.push({ y: y(v) })
+    }
+  }
+
+  const xticks = []
+  const tickStep = 3 * HOUR
+  const firstTick = Math.ceil(t0 / tickStep) * tickStep
+  for (let t = firstTick; t < t1; t += tickStep) {
+    if (t < t0) continue
+    const d = new Date(t)
+    xticks.push({
+      x: ((t - t0) / (t1 - t0)) * CHART_W,
+      left: ((t - t0) / (t1 - t0)) * 100,
+      label: `${String(d.getUTCHours()).padStart(2, '0')}:00`,
+      line: false,
+    })
+  }
+
+  return { bars, skyline: skyline.trim(), max, majors, minors, xticks, unit: '5min', series: [] }
 }
 
 /** X ticks for year/all: Monday boundaries up to a quarter, UTC month
@@ -327,7 +413,7 @@ export function fmtTick(t, span) {
   return d.toLocaleDateString(undefined, { year: 'numeric', timeZone: 'UTC' })
 }
 
-/** Y labels: integers when the step allows, one decimal for fractional steps. */
+/** Y labels use the same compact formatter as text labels. */
 export function fmtY(v) {
-  return Number.isInteger(v) ? String(v) : v.toFixed(1)
+  return formatCount(v)
 }

frontend/src/analytics/format.js

@@ -25,11 +25,44 @@ export const hostIP = (ip) => {
   }
 }
 
-/** Copy the full IP to the clipboard, ignoring failures. */
-export async function copyIp(ip) {
+function showCopiedFeedback(el) {
+  if (!el || typeof document === 'undefined') return
+  const popup = document.createElement('span')
+  popup.textContent = 'Copied!'
+  popup.className = 'copy-popup'
+  popup.style.cssText =
+    'position:absolute;bottom:calc(100% + 0.25rem);left:50%;' +
+    'transform:translateX(-50%);padding:0.15rem 0.4rem;' +
+    'background:var(--text, CanvasText);color:var(--bg, Canvas);' +
+    'border-radius:0.25rem;font-size:0.75rem;white-space:nowrap;' +
+    'pointer-events:none;z-index:10;'
+  el.classList.add('has-copy-popup')
+  el.appendChild(popup)
+  setTimeout(() => {
+    popup.remove()
+    el.classList.remove('has-copy-popup')
+  }, 1200)
+}
+
+/** Copy the full IP to the clipboard and show a brief "Copied!" popup. */
+export async function copyIp(ip, event) {
   if (!ip) return
+  const el = event?.currentTarget
   try {
     await navigator.clipboard.writeText(ip)
+    showCopiedFeedback(el)
+  } catch {
+    /* ignore */
+  }
+}
+
+/** Copy arbitrary text to the clipboard and show a brief "Copied!" popup. */
+export async function copyList(text, event) {
+  if (!text) return
+  const el = event?.currentTarget
+  try {
+    await navigator.clipboard.writeText(text)
+    showCopiedFeedback(el)
   } catch {
     /* ignore */
   }
@@ -44,6 +77,44 @@ export function calcTotalViews(views) {
   return n
 }
 
+// Very short reads are navigation/skims, not real reading time.
+export const MIN_READ_SECONDS = 10
+
+/** Average minutes per visit and average of per-article median read minutes. */
+export function calcReadStats(visits) {
+  const perArticle = {}
+  let totalVisitSeconds = 0
+  let visitCount = 0
+  for (const v of visits || []) {
+    const secs = Object.values(v.read || {}).filter((s) => s >= MIN_READ_SECONDS)
+    if (!secs.length) continue
+    visitCount++
+    totalVisitSeconds += secs.reduce((a, b) => a + b, 0)
+    for (const [path, s] of Object.entries(v.read || {})) {
+      if (s >= MIN_READ_SECONDS) {
+        ;(perArticle[path] || (perArticle[path] = [])).push(s)
+      }
+    }
+  }
+  const avgMinPerVisit = visitCount
+    ? Math.max(1, Math.round(totalVisitSeconds / visitCount / 60))
+    : 0
+
+  let articleMedianSum = 0
+  const articleCount = Object.keys(perArticle).length
+  for (const arr of Object.values(perArticle)) {
+    arr.sort((a, b) => a - b)
+    const mid = Math.floor(arr.length / 2)
+    const median = arr.length % 2 ? arr[mid] : (arr[mid - 1] + arr[mid]) / 2
+    articleMedianSum += Math.max(MIN_READ_SECONDS, median)
+  }
+  const avgArticleMedianMin = articleCount
+    ? Math.max(1, Math.round(articleMedianSum / articleCount / 60))
+    : 0
+
+  return { avgMinPerVisit, avgArticleMedianMin }
+}
+
 /** Build a path -> page title lookup from the site tree. */
 function buildTitleMap(pageTree) {
   const titles = new Map()
@@ -59,22 +130,22 @@ function buildTitleMap(pageTree) {
 
 /** Last path segment for display; front page becomes a house icon. */
 function slugOf(path) {
-  return path === '/' ? '🏠' : path.split('/').pop()
+  return path === '/' ? '🏠︎' : path.split('/').pop()
 }
 
-/** Host name of an external https origin, with scheme stripped. */
+/** Host name of an external https origin, with scheme and www. stripped. */
 function externalSlug(origin) {
   try {
-    return new URL(origin).host
+    return new URL(origin).host.replace(/^www\./, '')
   } catch {
-    return origin.replace(/^https?:\/\//, '')
+    return origin.replace(/^https?:\/\//, '').replace(/^www\./, '')
   }
 }
 
 /** Format one trail step: an internal page or an external https origin. */
 function stepOf(path, titles) {
   if (path?.startsWith('/')) {
-    return { path, slug: slugOf(path), title: titles.get(path) || '', external: false }
+    return { path, slug: slugOf(path), title: titles.get(path) || '', external: false, home: path === '/' }
   }
   if (path?.startsWith('https://')) {
     return {
@@ -140,6 +211,44 @@ export function formatWhenTooltip(ts) {
   return new Date(ts).toISOString().replace('T', ' ').replace('Z', ' UTC')
 }
 
+/** Full local timestamp for tooltips, e.g. "21 Aug 2026, 17:38:48". */
+export function formatWhenLocal(ts) {
+  return new Date(ts).toLocaleString('en-ie', {
+    year: 'numeric',
+    month: 'short',
+    day: 'numeric',
+    hour: '2-digit',
+    minute: '2-digit',
+    second: '2-digit',
+  })
+}
+
+/** Preserve locale case with the region/country subtag upper-cased. */
+export function formatLang(value) {
+  if (!value || value === '—') return value
+  const parts = value.split('-')
+  if (parts.length > 1) {
+    parts[parts.length - 1] = parts[parts.length - 1].toUpperCase()
+  }
+  return parts.join('-')
+}
+
+/** ISO 8601 UTC timestamp without subseconds, e.g. "2026-08-21T00:20:48Z". */
+export function formatWhenIso(ts) {
+  return `${new Date(ts).toISOString().split('.')[0]}Z`
+}
+
+/**
+ * Compact visitor counts: plain below 1k, then 1.2k / 10k / 1.2M.
+ * Truncated, not rounded.
+ */
+export function formatCount(n) {
+  if (n < 1000) return String(n)
+  if (n < 10000) return `${Math.trunc(n / 1000)}.${Math.trunc((n % 1000) / 100)}k`
+  if (n < 1_000_000) return `${Math.trunc(n / 1000)}k`
+  return `${Math.trunc(n / 1_000_000)}.${Math.trunc((n % 1_000_000) / 100_000)}M`
+}
+
 /**
  * Format recent visits for display, newest first. Each step is a linked slug
  * pointing to its article; external referers/origins are shown as their
@@ -196,32 +305,52 @@ export function formatCounts(entries) {
 
 /**
  * Count distinct User-Agent strings among crawler hits, most common first.
- * Returns an array of [ua, count] pairs.
+ * Returns an array of [ua, count] pairs.  ``clients`` maps client hashes to
+ * client records.
  */
-export function countCrawlerUas(crawlers) {
+export function countCrawlerUas(crawlers, clients) {
   const counts = {}
   for (const c of crawlers || []) {
-    const value = c.ua_pretty || c.ua || '(no UA)'
+    const client = (clients || {})[c.client] || {}
+    const value = client.ua_pretty || client.ua || '(no UA)'
     counts[value] = (counts[value] || 0) + 1
   }
   return Object.entries(counts).sort((a, b) => b[1] - a[1])
 }
 
 /**
- * Group raw crawler hits by the same (ip, ua) pair we use to tell a real
- * visitor from a crawler, and format each group as a row showing every
- * internal page that crawler visited.  Rows are sorted by total hits,
- * most active crawler first, rather than by most recent hit.
+ * Reduce a reverse-DNS hostname to its right-most components that fit
+ * within ``limit`` characters.  This keeps the meaningful main domain
+ * while avoiding absurdly long subdomains like ``xxx.yyy.zzz...provider.net``.
  */
-export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
+export function mainDomain(host, limit = 24) {
+  if (!host) return host
+  const labels = host.split('.').filter(Boolean)
+  if (!labels.length) return host
+  const parts = [labels.pop()]
+  while (labels.length) {
+    const next = labels[labels.length - 1]
+    const candidate = `${next}.${parts.join('.')}`
+    if (candidate.length > limit) break
+    parts.unshift(labels.pop())
+  }
+  return parts.join('.')
+}
+
+/**
+ * Group raw crawler hits by client hash and format each group as a row showing
+ * every internal page that crawler visited.  Rows are sorted by total hits,
+ * most active crawler first, rather than by most recent hit.
+ * ``clients`` maps client hashes to client records.
+ */
+export function formatCrawlerRows(crawlers, clients, pageTree, now = Date.now()) {
   const titles = buildTitleMap(pageTree)
   const groups = new Map()
   for (const c of crawlers || []) {
-    const key = `${c.ip}\0${c.ua}`
-    const g = groups.get(key) || {
-      ip: c.ip || '',
-      ua: c.ua_pretty || c.ua || '—',
-      uaRaw: c.ua || '',
+    const client = (clients || {})[c.client] || {}
+    const g = groups.get(c.client) || {
+      clientHash: c.client,
+      client,
       lastStart: 0,
       pages: new Map(),
     }
@@ -230,7 +359,7 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
     if (c.entry?.startsWith('/')) {
       g.pages.set(c.entry, (g.pages.get(c.entry) || 0) + 1)
     }
-    groups.set(key, g)
+    groups.set(c.client, g)
   }
   const totalHits = (g) => {
     let n = 0
@@ -240,23 +369,121 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
   return [...groups.values()]
     .sort((a, b) => totalHits(b) - totalHits(a) || b.lastStart - a.lastStart)
     .slice(0, 10)
-    .map((g) => ({
-      when: formatWhen(g.lastStart, now),
-      whenTooltip: formatWhenTooltip(g.lastStart),
-      pages: [...g.pages.entries()]
-        .sort((a, b) => b[1] - a[1])
-        .map(([path, count]) => ({
-          path,
-          slug: slugOf(path),
-          title: titles.get(path) || '',
-          count,
+    .map((g) => {
+      const client = g.client || {}
+      const host = client.host || ''
+      const isHost = !!host
+      return {
+        lastSeen: formatWhen(g.lastStart, now),
+        lastSeenIso: formatWhenIso(g.lastStart),
+        lastSeenLocal: formatWhenLocal(g.lastStart),
+        pages: [...g.pages.entries()]
+          .sort((a, b) => b[1] - a[1])
+          .map(([path, count]) => ({ ...stepOf(path, titles), count })),
+        ip: client.ip || '',
+        ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip) || client.ip || '—',
+        isHost,
+        ua: client.ua_pretty || client.ua || '—',
+        uaRaw: client.ua || '',
+        lang: client.lang || '—',
+        langDisplay: formatLang(client.lang),
+        country: client.country || '—',
+        city: client.city || '—',
+        total: totalHits(g),
+      }
+    })
+}
+
+/**
+ * Group abuse hits by IP and format each group as a row with the full paths
+ * probed.  Identical paths are collapsed into one entry with their hit count.
+ * Flagged paths (the ones that triggered abuse classification) are lifted to
+ * the top, followed by other 404s, then document GETs from the abuser.  Within
+ * each category paths are sorted by count descending, then earliest first.
+ * Rows are sorted by most recent hit first.  Visitor metadata comes from the
+ * latest client hash seen for the IP; ``clientCount`` tells the visitor cell
+ * how many distinct client variations the IP produced.  Paths are shown
+ * verbatim (query string included), not resolved against the page tree.
+ * ``clients`` maps client hashes to client records.
+ */
+export function formatAbuseRows(abuse, clients, now = Date.now()) {
+  const groups = new Map()
+  for (const a of abuse || []) {
+    const client = (clients || {})[a.client] || {}
+    const ip = client.ip || ''
+    const g = groups.get(ip) || {
+      ip,
+      pathCounts: new Map(),
+      clientHashes: new Set(),
+      lastStart: 0,
+      lastClient: a.client,
+    }
+    const start = new Date(a.start).getTime()
+    if (start > g.lastStart) {
+      g.lastStart = start
+      g.lastClient = a.client
+    }
+    const path = a.path || ''
+    const existing = g.pathCounts.get(path) || {
+      path,
+      count: 0,
+      firstStart: start,
+      flag: a.flag || false,
+      is_404: a.is_404 || false,
+    }
+    existing.count += 1
+    if (start < existing.firstStart) existing.firstStart = start
+    if (a.flag) existing.flag = true
+    if (!a.is_404) existing.is_404 = false
+    g.pathCounts.set(path, existing)
+    g.clientHashes.add(a.client)
+    groups.set(ip, g)
+  }
+  const totalHits = (g) => {
+    let n = 0
+    for (const p of g.pathCounts.values()) n += p.count
+    return n
+  }
+  return [...groups.values()]
+    .sort((a, b) => b.lastStart - a.lastStart)
+    .slice(0, 10)
+    .map((g) => {
+      const pathCategory = (p) => (p.flag ? 0 : p.is_404 ? 1 : 2)
+      const paths = [...g.pathCounts.values()].sort(
+        (a, b) =>
+          pathCategory(a) - pathCategory(b) ||
+          b.count - a.count ||
+          a.firstStart - b.firstStart,
+      )
+      const client = (clients || {})[g.lastClient] || {}
+      const host = client.host || ''
+      const isHost = !!host
+      return {
+        lastSeen: formatWhen(g.lastStart, now),
+        lastSeenIso: formatWhenIso(g.lastStart),
+        lastSeenLocal: formatWhenLocal(g.lastStart),
+        paths: paths.map((p) => ({
+          path: p.path,
+          count: p.count,
+          flag: p.flag,
+          is_404: p.is_404,
         })),
-      ip: g.ip,
-      ipDisplay: hostIP(g.ip) || g.ip || '—',
-      ua: g.ua,
-      uaRaw: g.uaRaw,
-      total: totalHits(g),
-    }))
+        allPaths: paths
+          .map((p) => (p.count > 1 ? `${p.count}× ${p.path}` : p.path))
+          .join('\n'),
+        clientCount: g.clientHashes.size,
+        ip: client.ip || g.ip,
+        ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip || g.ip) || client.ip || g.ip || '—',
+        isHost,
+        ua: client.ua_pretty || client.ua || '—',
+        uaRaw: client.ua || '',
+        lang: client.lang || '—',
+        langDisplay: formatLang(client.lang),
+        country: client.country || '—',
+        city: client.city || '—',
+        total: totalHits(g),
+      }
+    })
 }
 
 /**
@@ -264,31 +491,42 @@ export function formatCrawlerRows(crawlers, pageTree, now = Date.now()) {
  * with display strings; missing values become "—".  ``trail`` starts with the
  * external referer (when present), then the entry page and any further internal
  * pages or external exit origins. Only the 20 most recent visits are shown.
+ * ``clients`` maps client hashes to client records.
  */
-export function formatVisitRows(visits, pageTree, now = Date.now()) {
+export function formatVisitRows(visits, clients, pageTree, now = Date.now()) {
   const titles = buildTitleMap(pageTree)
   return [...(visits || [])].reverse().slice(0, 20).map((v) => {
-    const trail = [v.referer, v.entry, ...(v.trail || [])]
+    const client = (clients || {})[v.client] || {}
+    const trail = [v.entry, ...(v.trail || [])]
       .map((p) => stepOf(p, titles))
       .filter(Boolean)
-    const utm = Object.entries(v.utm || {})
+    const utmKeys = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content']
+    const utmValues = utmKeys.map((k) => (v.utm || {})[k]).filter(Boolean)
+    const utm = utmValues.length ? utmValues.join(' · ') : ''
+    const utmTitle = Object.entries(v.utm || {})
       .map(([k, value]) => `${k}=${value}`)
       .join(', ')
     const dash = (s) => (s || '—')
+    const host = client.host || ''
+    const isHost = !!host
     return {
-      when: formatWhen(v.start, now),
-      whenTooltip: formatWhenTooltip(v.start),
+      lastSeen: formatWhen(v.start, now),
+      lastSeenIso: formatWhenIso(v.start),
+      lastSeenLocal: formatWhenLocal(v.start),
+      langDisplay: formatLang(client.lang),
       trail,
+      refererStep: stepOf(v.referer, titles),
       referer: dash(v.referer),
-      ip: v.ip || '',
-      ipDisplay: v.host || hostIP(v.ip) || v.ip || '—',
-      host: dash(v.host),
-      lang: dash(v.lang),
-      country: dash(v.country),
-      city: dash(v.city),
-      ua: v.ua_pretty || v.ua || '—',
-      uaRaw: v.ua || '',
+      ip: client.ip || '',
+      ipDisplay: isHost ? mainDomain(host) : hostIP(client.ip) || client.ip || '—',
+      isHost,
+      lang: dash(client.lang),
+      country: dash(client.country),
+      city: dash(client.city),
+      ua: client.ua_pretty || client.ua || '—',
+      uaRaw: client.ua || '',
       utm: utm || '—',
+      utmTitle,
     }
   })
 }

frontend/src/analytics/time.js

@@ -13,6 +13,7 @@ export const DAY = 86400e3
 export const WEEK = 7 * DAY
 
 export const RANGES = {
+  day: { label: 'day', span: DAY, bucket: MIN5 },
   week: { label: 'week' },
   month: { label: 'month', span: 30 * DAY, bucket: 6 * HOUR },
   year: { label: 'year', span: 365 * DAY, bucket: DAY },
@@ -117,11 +118,36 @@ export function rollingSeries(buckets, rangeKey) {
   }
 }
 
-/** Dispatch to weekly or rolling series based on the selected range. */
+/**
+ * Day view: raw 5-minute bucket counts for the current 24-hour window.
+ * No smoothing or rate conversion is applied; counts are used as-is.
+ */
+export function daySeries(buckets) {
+  const raw = rawTimes(buckets)
+  const now = Date.now()
+  const { span, bucket } = RANGES.day
+  const t1 = Math.floor(now / bucket) * bucket + bucket
+  const t0 = t1 - span
+  const points = []
+  for (let t = t0; t < t1; t += bucket) {
+    points.push({ t, count: raw[t] || 0 })
+  }
+  return {
+    series: [{ points, label: '', opacity: 1, area: false }],
+    t0,
+    t1,
+    rate: 1,
+    binMinutes: bucket / 60e3,
+    unitMinutes: bucket / 60e3,
+    unit: '5min',
+  }
+}
+
+/** Dispatch to daily, weekly or rolling series based on the selected range. */
 export function makeSeries(buckets, rangeKey) {
-  return rangeKey === 'week'
-    ? weeklySeries(buckets)
-    : rollingSeries(buckets, rangeKey)
+  if (rangeKey === 'day') return daySeries(buckets)
+  if (rangeKey === 'week') return weeklySeries(buckets)
+  return rollingSeries(buckets, rangeKey)
 }
 
 /**

frontend/src/analytics/transitions.js

@@ -13,32 +13,39 @@
  * connections. Animated beads flow along every edge in each direction,
  * emitted at time intervals inversely proportional (linear) to the
  * directional count.
- * External referers appear as nodes in a row above the map, external exits
- * as full-size nodes just outside their source page, angled away from the
- * center. Each distinct full exit URL is its own node. Self-loops (reload
- * pings) are skipped.
+ * External sources appear as nodes in a row above the map. Sources are
+ * identified from visit records in this order: utm_campaign, utm_source,
+ * referer, then other utm_* tags. Visits with a UTM tag are grouped under
+ * that tag's value, not under the referer domain. A UTM source node only
+ * becomes a clickable link when every visit carrying that UTM tag came
+ * from the same referer. External exits are full-size nodes just outside
+ * their source page, angled away from the center. Each distinct full exit
+ * URL is its own node. Self-loops (reload pings) are skipped.
  */
 
+import { MIN_READ_SECONDS } from './format.js'
+
 export const TNODE_R = 34 // node circles hold the slug and the view count
 export const EXT_R = 34 // external referer/exit nodes use the same full size
 
 // Edge width (half-width of the thin middle) grows logarithmically with
-// the count, anchored so a single recorded transition renders as a ~1 px
-// line. There is no cap — growth is slow enough that even very hot
-// connections stay reasonable. Connections carrying less than
-// PRUNE_FRACTION of the total traffic are not drawn at all (this also
-// keeps the number of drawn connections under ~100).
-const WMID_MIN = 0.5
-const WIDTH_GROWTH = 1.5
+// the count. The constants are scaled down by ~10× so busy ranges (day,
+// year) do not overwhelm the graph with fat connectors. A single recorded
+// transition still renders as a faint ~0.4 px line. Connections carrying
+// less than PRUNE_FRACTION of the total traffic are not drawn at all (this
+// also keeps the graph under ~100 connections).
+const WMID_MIN = 0.2
+const WIDTH_GROWTH = 0.15
 const PRUNE_FRACTION = 0.01
 
 // Beads: each edge direction emits beads at count * BEAD_RATE beads per
-// second (linear in the count). The component simulates every bead
-// independently in JS at BEAD_SPEED along the edge, with no limit on
+// second (linear in the count). The rate is reduced ~10× across all time
+// scales to keep the animation lightweight. The component simulates every
+// bead independently in JS at BEAD_SPEED along the edge, with no limit on
 // beads in flight.
 export const BEAD_SPEED = 180 // svg units per second
 export const BEAD_R = 2.2
-const BEAD_RATE = 0.12 // beads per second per recorded transition
+const BEAD_RATE = 0.012 // beads per second per recorded transition
 const FLOW_OFFSET = 3 // lane offset to the right of the travel direction
 
 const MAX_EXT_IN = 8 // referer nodes in the top row
@@ -84,30 +91,31 @@ function collectInternalTransitions(transitions) {
   return internal
 }
 
-/** Short display label for an external origin (protocol stripped). */
+/** Domain-only label for an external origin (path and www. removed). */
 function extLabel(ext) {
-  const s = ext.replace(/^https?:\/\//, '')
-  return s.length > 11 ? `${s.slice(0, 10)}…` : s
+  try {
+    const host = new URL(ext).hostname.replace(/^www\./, '')
+    return host.length > 25 ? `${host.slice(0, 24)}…` : host
+  } catch {
+    const s = ext.replace(/^https?:\/\//, '').replace(/^www\./, '').split('/')[0]
+    return s.length > 25 ? `${s.slice(0, 24)}…` : s
+  }
 }
 
 /**
- * Collect external transitions: referer origin -> entry page (incoming) and
- * page -> exit origin (outgoing). Aggregated per (origin, page) pair, with
- * separate directional counts. "(direct)" entries are not links and skipped.
+ * Collect outgoing external transitions: page path -> full exit URL.
+ * Aggregated per (URL, page) pair. Incoming external links are now derived
+ * from visit records (which carry UTM tags), so only exits remain here.
  */
-function collectExternalPairs(transitions) {
-  const pairs = new Map() // `${ext} ${page}` -> {ext, page, in, out}
+function collectExitPairs(transitions) {
+  const pairs = new Map() // `${ext} ${page}` -> {ext, page, out}
   for (const [fr, tos] of Object.entries(transitions || {})) {
+    if (!fr.startsWith('/')) continue // ignore external -> anything
     for (const [to, count] of Object.entries(tos)) {
-      const frExt = !fr.startsWith('/')
-      const toExt = !to.startsWith('/')
-      if (frExt === toExt) continue // internal-internal or ext-ext
-      const ext = frExt ? fr : to
-      const page = frExt ? to : fr
-      if (!ext.startsWith('http')) continue
-      const k = `${ext} ${page}`
-      const p = pairs.get(k) || { ext, page, in: 0, out: 0 }
-      p[frExt ? 'in' : 'out'] += count
+      if (!to.startsWith('http')) continue
+      const k = `${to} ${fr}`
+      const p = pairs.get(k) || { ext: to, page: fr, out: 0 }
+      p.out += count
       pairs.set(k, p)
     }
   }
@@ -175,8 +183,29 @@ function layoutAngles(root, unit, weight) {
   }
 }
 
+/** Compute median reading time per article in whole minutes. */
+function buildReadMinutes(visits) {
+  const times = {}
+  for (const v of visits || []) {
+    for (const [path, sec] of Object.entries(v.read || {})) {
+      if (sec >= MIN_READ_SECONDS) {
+        ;(times[path] || (times[path] = [])).push(sec)
+      }
+    }
+  }
+  const minutes = {}
+  for (const [path, arr] of Object.entries(times)) {
+    arr.sort((a, b) => a - b)
+    const mid = Math.floor(arr.length / 2)
+    const median =
+      arr.length % 2 ? arr[mid] : (arr[mid - 1] + arr[mid]) / 2
+    minutes[path] = Math.max(1, Math.round(median / 60))
+  }
+  return minutes
+}
+
 /** Compute radial positions, view counts and labels for each node. */
-function positionNodes(nodes, maxDepth, unit, viewsData, titles) {
+function positionNodes(nodes, maxDepth, unit, viewsData, titles, readMinutes) {
   // Constant radial gap between rings, equal to the arc spacing of nodes
   // along a ring: leaf arc = unit * GAP, so GAP scales up with `unit` on
   // sparse trees (where closing the circle forces wider arcs) and with
@@ -196,10 +225,14 @@ function positionNodes(nodes, maxDepth, unit, viewsData, titles) {
     n.x = Math.cos(n.angle) * r
     n.y = Math.sin(n.angle) * r
     n.views = viewCount(n.path)
+    n.readMin = readMinutes[n.path] || 0
     // Slug inside the circle; full title goes on the link title attribute.
-    const slug = n.path === '/' ? '🏠' : n.path.split('/').pop()
-    n.label = slug.length > 11 ? `${slug.slice(0, 10)}…` : slug
+    const slug = n.path === '/' ? '🏠︎' : n.path.split('/').pop()
+    n.label = slug.length > 16 ? `${slug.slice(0, 15)}…` : slug
     n.title = titles.get(n.path) || ''
+    // Category (non-leaf) pages with no views in this window are left
+    // blank to keep the layout, but their circle/label is not drawn.
+    n.hidden = n.children.length > 0 && n.views === 0
   }
 
   return { radius, GAP }
@@ -279,7 +312,7 @@ const fmtPt = (p) => `${p[0].toFixed(2)} ${p[1].toFixed(2)}`
  * `wMid` is the half-width of the thin middle (already strength-scaled by
  * the caller); `ra`/`rb` are the radii of the node circles each end wraps.
  */
-function buildRibbon(a, b, ab, ba, wMid, ra = TNODE_R, rb = TNODE_R) {
+function buildRibbon(a, b, ab, ba, wMid, ra = TNODE_R, rb = TNODE_R, external = false) {
   const count = ab + ba
   const len = Math.hypot(b.x - a.x, b.y - a.y) || 1
   const ux = (b.x - a.x) / len
@@ -387,6 +420,7 @@ function buildRibbon(a, b, ab, ba, wMid, ra = TNODE_R, rb = TNODE_R) {
   return {
     d,
     title: `${a.path} ↔ ${b.path}: ${count} (${ab} / ${ba})`,
+    external,
   }
 }
 
@@ -401,7 +435,7 @@ function buildRibbon(a, b, ab, ba, wMid, ra = TNODE_R, rb = TNODE_R) {
  * edge run on parallel lanes instead of colliding. The component turns
  * these into independently simulated beads.
  */
-function buildFlows(a, b, ra, rb, ab, ba) {
+function buildFlows(a, b, ra, rb, ab, ba, visualScale = 1) {
   const len = Math.hypot(b.x - a.x, b.y - a.y) || 1
   const ux = (b.x - a.x) / len
   const uy = (b.y - a.y) / len
@@ -422,7 +456,7 @@ function buildFlows(a, b, ra, rb, ab, ba) {
       x2: a.x + toT * ux + s * rx,
       y2: a.y + toT * uy + s * ry,
       len: span,
-      interval: 1 / (count * BEAD_RATE),
+      interval: 1 / (count * BEAD_RATE * visualScale),
     }
   }
   const flows = []
@@ -437,14 +471,17 @@ function buildFlows(a, b, ra, rb, ab, ba) {
  * Absolute on purpose — cool routes stay visible regardless of how hot
  * the hottest connection is.
  */
-const scaledWidth = (count) => WMID_MIN + WIDTH_GROWTH * Math.log1p(count - 1)
+const scaledWidth = (count) => {
+  if (count <= 0) return 0
+  return WMID_MIN + WIDTH_GROWTH * Math.log1p(count - 1)
+}
 
 /**
  * Build ribbon edges and bead flows for every aggregated page-to-page
  * pair. Pairs carrying less than PRUNE_FRACTION of the total internal
  * traffic are pruned (this naturally bounds the graph to ~100 edges).
  */
-function buildInternalEdges(pairs, byPath) {
+function buildInternalEdges(pairs, byPath, visualScale = 1) {
   let total = 0
   for (const [, [ab, ba]] of pairs) total += ab + ba
   const minCount = total * PRUNE_FRACTION
@@ -456,8 +493,10 @@ function buildInternalEdges(pairs, byPath) {
     const [pf, pt] = k.split(' ')
     const a = byPath.get(pf)
     const b = byPath.get(pt)
-    edges.push(buildRibbon(a, b, ab, ba, scaledWidth(ab + ba)))
-    flows.push(...buildFlows(a, b, TNODE_R, TNODE_R, ab, ba))
+    const wMid = scaledWidth((ab + ba) * visualScale)
+    if (wMid <= 0) continue
+    edges.push(buildRibbon(a, b, ab, ba, wMid))
+    flows.push(...buildFlows(a, b, TNODE_R, TNODE_R, ab, ba, visualScale))
   }
   return { edges, flows }
 }
@@ -498,42 +537,120 @@ export function filterViewsByRange(views, t0, t1) {
   return filtered
 }
 
+/** Keep only visits whose start time falls inside [t0, t1). */
+export function filterVisitsByRange(visits, t0, t1) {
+  const out = []
+  for (const v of visits || []) {
+    const t = Date.parse(v.start)
+    if ((t0 == null || t >= t0) && (t1 == null || t < t1)) out.push(v)
+  }
+  return out
+}
+
+const UTM_PRIORITY = ['utm_campaign', 'utm_source']
+const UTM_FALLBACK = ['utm_medium', 'utm_content', 'utm_term', 'utm_id']
+
+/** Identify the source of a visit according to the requested priority. */
+function identifySource(visit) {
+  const utm = visit.utm || {}
+  for (const k of UTM_PRIORITY) {
+    const v = utm[k]
+    if (v) return { value: v, isUtm: true }
+  }
+  if (visit.referer?.startsWith('http')) {
+    return { value: visit.referer, isUtm: false }
+  }
+  for (const k of UTM_FALLBACK) {
+    const v = utm[k]
+    if (v) return { value: v, isUtm: true }
+  }
+  return null
+}
+
 /**
- * Place external referer and exit nodes and build their edges and bead
+ * Collect source -> entry page pairs from visit records. Sources are
+ * identified by UTM campaign/source (then referer, then other UTM tags).
+ * A UTM source only gets a link href when every visit using that source
+ * came from the same referer; referer sources always link to their origin.
+ */
+function collectSourcePairs(visits) {
+  const groups = new Map() // `${source}\0${page}` -> pair
+  for (const v of visits || []) {
+    const src = identifySource(v)
+    if (!src) continue
+    const k = `${src.value}\0${v.entry}`
+    const p = groups.get(k) || {
+      source: src.value,
+      page: v.entry,
+      in: 0,
+      refs: new Set(),
+      missingRef: false,
+      href: null,
+      isUtm: src.isUtm,
+    }
+    p.in += 1
+    if (v.referer?.startsWith('http')) {
+      p.refs.add(v.referer)
+    } else {
+      p.missingRef = true
+    }
+    groups.set(k, p)
+  }
+  for (const p of groups.values()) {
+    if (p.isUtm && !p.missingRef && p.refs.size === 1) {
+      const ref = [...p.refs][0]
+      if (ref.startsWith('http')) p.href = ref
+    } else if (!p.isUtm && p.source.startsWith('http')) {
+      p.href = p.source
+    }
+  }
+  return [...groups.values()]
+}
+
+/**
+ * Place external source and exit nodes and build their edges and bead
  * flows.
- * Referers (incoming links) form a row centered above the map, hottest
- * first; exits sit just outside their source page, fanned away from the
- * center and nudged outwards until they no longer overlap any node.
+ * Sources (incoming links) are derived from visit UTM/referer data and form
+ * a row centered above the map, hottest first; exits come from the
+ * transition matrix and sit just outside their source page.
  * Widths and pruning use the same log scale and traffic-share rule as
  * internal connections.
  */
-function buildExternal(external, byPath, radius, innerBounds) {
+function buildExternal({ sources, exits }, byPath, radius, innerBounds, visualScale = 1) {
   const extNodes = []
   const edges = []
   const flows = []
   let extTotal = 0
-  for (const p of external) extTotal += p.in + p.out
+  for (const p of sources) extTotal += p.in
+  for (const p of exits) extTotal += p.out
   const minCount = extTotal * PRUNE_FRACTION
-  const live = external.filter((p) => byPath.has(p.page))
-  if (!live.length) return { extNodes, edges, flows }
+  const liveSources = sources.filter((p) => byPath.has(p.page))
+  const liveExits = exits.filter((p) => byPath.has(p.page))
+  if (!liveSources.length && !liveExits.length) return { extNodes, edges, flows }
 
-  const width = scaledWidth
+  const width = (count) => scaledWidth(count * visualScale)
 
   const overlaps = (x, y, r) =>
     [...byPath.values(), ...extNodes].some(
       (n) => Math.hypot(n.x - x, n.y - y) < (n.r ?? TNODE_R) + r + 10,
     )
 
-  // Incoming: one referer node per origin, in a row centered above the
-  // map, with an edge to each page that origin led to.
-  const byExt = new Map() // ext -> pairs, sorted by total incoming count
-  for (const p of live.filter((p) => p.in >= minCount)) {
-    const g = byExt.get(p.ext) || []
+  // Incoming: one source node per identified source, in a row centered
+  // above the map, with an edge to each page that source led to.
+  const bySource = new Map() // source -> pairs, sorted by total incoming count
+  for (const p of liveSources.filter((p) => p.in >= minCount)) {
+    const g = bySource.get(p.source) || []
     g.push(p)
-    byExt.set(p.ext, g)
+    bySource.set(p.source, g)
   }
-  const origins = [...byExt]
-    .map(([ext, ps]) => ({ ext, ps, total: ps.reduce((s, p) => s + p.in, 0) }))
+  const origins = [...bySource]
+    .map(([source, ps]) => ({
+      source,
+      ps,
+      total: ps.reduce((s, p) => s + p.in, 0),
+      href: ps[0].href,
+      isUtm: ps[0].isUtm,
+    }))
     .sort((a, b) => b.total - a.total)
     .slice(0, MAX_EXT_IN)
   if (origins.length) {
@@ -541,14 +658,25 @@ function buildExternal(external, byPath, radius, innerBounds) {
     const y = innerBounds.y0 - TNODE_R - 64
     const spacing = 2 * EXT_R + 44
     const x0 = cx - ((origins.length - 1) * spacing) / 2
-    origins.forEach(({ ext, ps }, i) => {
-      const total = ps.reduce((s, p) => s + p.in, 0)
-      const xn = { path: ext, label: extLabel(ext), x: x0 + i * spacing, y, r: EXT_R, count: total, kind: 'source' }
+    origins.forEach(({ source, ps, total, href, isUtm }, i) => {
+      const label = isUtm ? source : extLabel(source)
+      const xn = {
+        path: source,
+        href,
+        label: label.length > 25 ? `${label.slice(0, 24)}…` : label,
+        x: x0 + i * spacing,
+        y,
+        r: EXT_R,
+        count: total,
+        kind: 'source',
+      }
       extNodes.push(xn)
       for (const p of ps) {
         const page = byPath.get(p.page)
-        edges.push(buildRibbon(xn, page, p.in, 0, width(p.in), EXT_R, TNODE_R))
-        flows.push(...buildFlows(xn, page, EXT_R, TNODE_R, p.in, 0))
+        const wMid = width(p.in)
+        if (wMid <= 0) continue
+        edges.push(buildRibbon(xn, page, p.in, 0, wMid, EXT_R, TNODE_R, true))
+        flows.push(...buildFlows(xn, page, EXT_R, TNODE_R, p.in, 0, visualScale))
       }
     })
   }
@@ -558,7 +686,7 @@ function buildExternal(external, byPath, radius, innerBounds) {
   // (same radial spacing internal rings use), fanned around the source angle,
   // and shows the total count across all pages that link to that URL.
   const GAP = radius(1) - radius(0)
-  const outgoing = live.filter((p) => p.out >= minCount)
+  const outgoing = liveExits.filter((p) => p.out >= minCount)
     .sort((a, b) => b.out - a.out)
   const perPage = new Map()
   const selected = []
@@ -588,13 +716,24 @@ function buildExternal(external, byPath, radius, innerBounds) {
         x = page.x + Math.cos(ang) * dist
         y = page.y + Math.sin(ang) * dist
       }
-      xn = { path: p.ext, label: extLabel(p.ext), x, y, r: EXT_R, count: 0, kind: 'exit' }
+      xn = {
+        path: p.ext,
+        href: p.ext,
+        label: extLabel(p.ext),
+        x,
+        y,
+        r: EXT_R,
+        count: 0,
+        kind: 'exit',
+      }
       exitNodes.set(p.ext, xn)
       extNodes.push(xn)
     }
     xn.count += p.out
-    edges.push(buildRibbon(page, xn, p.out, 0, width(p.out), TNODE_R, EXT_R))
-    flows.push(...buildFlows(page, xn, TNODE_R, EXT_R, p.out, 0))
+    const wMid = width(p.out)
+    if (wMid <= 0) continue
+    edges.push(buildRibbon(page, xn, p.out, 0, wMid, TNODE_R, EXT_R, true))
+    flows.push(...buildFlows(page, xn, TNODE_R, EXT_R, p.out, 0, visualScale))
   }
 
   return { extNodes, edges, flows }
@@ -605,11 +744,13 @@ function buildExternal(external, byPath, radius, innerBounds) {
  * Returns { nodes, edges, flows, extNodes, arcs, bounds } or null when
  * there is nothing to show.
  */
-export function buildTransitionGraph(data, pageTree) {
+export function buildTransitionGraph(data, pageTree, visits = [], visualScale = 1) {
   const internal = collectInternalTransitions(data?.transitions)
-  const external = collectExternalPairs(data?.transitions)
+  const sources = collectSourcePairs(visits)
+  const exits = collectExitPairs(data?.transitions)
   const navOrder = buildNavigationOrder(pageTree)
   const titles = buildTitleMap(pageTree)
+  const readMinutes = buildReadMinutes(visits)
 
   if (!internal.length && !navOrder.size) return null
 
@@ -619,10 +760,10 @@ export function buildTransitionGraph(data, pageTree) {
   layoutAngles(root, unit, weightFn)
 
   const maxDepth = Math.max(1, ...nodes.map((n) => n.depth))
-  const { radius } = positionNodes(nodes, maxDepth, unit, data?.views, titles)
+  const { radius } = positionNodes(nodes, maxDepth, unit, data?.views, titles, readMinutes)
   const arcs = buildFamilyArcs(nodes, radius)
   const pairs = aggregatePairs(internal)
-  const { edges, flows } = buildInternalEdges(pairs, byPath)
+  const { edges, flows } = buildInternalEdges(pairs, byPath, visualScale)
 
   // Tight bounding box of the actual page nodes; family ring arcs can sweep
   // outside the node circle (e.g. a large arc between two siblings on the
@@ -646,7 +787,7 @@ export function buildTransitionGraph(data, pageTree) {
     bounds.y1 = Math.max(bounds.y1, b.y1)
   }
 
-  const ext = buildExternal(external, byPath, radius, bounds)
+  const ext = buildExternal({ sources, exits }, byPath, radius, bounds, visualScale)
   for (const xn of ext.extNodes) {
     bounds.x0 = Math.min(bounds.x0, xn.x - xn.r - pad)
     bounds.y0 = Math.min(bounds.y0, xn.y - xn.r - pad)

frontend/src/pagerite.js

@@ -338,29 +338,107 @@ import "overlayscrollbars/overlayscrollbars.css";
   }
 
   // --- Analytics pings ---------------------------------------------------
-  // Fire-and-forget POST /_a {fr, to}: on the initial page load (starts the
-  // visit — the server counts nothing from the document GET alone), for
-  // internal fetch-navigations and for external https exits. Excluded:
-  // back/forward (popstate never pings) and everything while we know the
-  // user is an admin — but only when SSO is actually in use; with no auth
-  // (dev/test) "admin" is everyone's state and nothing would be recorded —
-  // or has the editor open (admin noise, not visits). The analytics page
-  // itself (/_a) is also excluded even though fetch-navigation treats it like
-  // a normal article.
+  // Fire-and-forget POST /_a {fr, to, read}: on the initial page load
+  // (starts the visit — the server counts nothing from the document GET
+  // alone), for internal fetch-navigations, for external https exits, and
+  // on window close. ``read`` is the active time (ms) spent on ``fr``.
+  // Reading time pauses after 1 minute of inactivity and resumes on the
+  // next mouse/touch/scroll/keyboard event.
+  // Excluded: back/forward (popstate never pings), everything while the
+  // editor is open (body.editing — admin noise, not visits), and the
+  // analytics page itself (/_a), even though fetch-navigation treats it
+  // like a normal article.
+  // Admins (when SSO is actually in use — with no auth proxy "admin" is
+  // everyone's state) ping normally but with hide=1: the server then
+  // records nothing and scrubs any session the same browser accumulated
+  // before logging in, so admins never show up as visits or crawlers.
   // See docs/analytics.md.
-  function ping(to, fr = currentPath) {
-    if ((ssoAvailable && isAdmin) || document.body.classList.contains("editing")
-        || to === "/_a" || fr === "/_a") return;
+  function ping(to, fr = currentPath, read = 0) {
+    if (document.body.classList.contains("editing")) return;
+    if ((to && to === "/_a") || fr === "/_a") return;
+    const hide = ssoAvailable && isAdmin ? 1 : 0;
+    const body = JSON.stringify({
+      fr, to, hide,
+      read: Math.max(0, Math.round(read / 1000)),
+    });
     try {
       fetch("/_a", {
         method: "POST",
         keepalive: true,
         headers: { "content-type": "application/json" },
-        body: JSON.stringify({ fr, to }),
+        body,
       });
     } catch { /* analytics must never break navigation */ }
   }
 
+  // Active reading time for the current page. The clock stops after 1 minute
+  // without activity and restarts on the next mouse/touch/scroll/keyboard
+  // event.
+  const INACTIVE_MS = 60_000;
+  let readStart = performance.now();
+  let readElapsed = 0;
+  let reading = true;
+  let readInactivityTimer = null;
+  let closePingedFor = null;
+
+  function markReadActivity() {
+    if (!reading) {
+      reading = true;
+      readStart = performance.now();
+    }
+    clearTimeout(readInactivityTimer);
+    readInactivityTimer = setTimeout(() => {
+      if (reading) {
+        readElapsed += performance.now() - readStart;
+        reading = false;
+      }
+    }, INACTIVE_MS);
+  }
+
+  function takeReadTime() {
+    if (reading) {
+      readElapsed += performance.now() - readStart;
+      readStart = performance.now();
+    }
+    const ms = Math.max(0, Math.round(readElapsed));
+    readElapsed = 0;
+    return ms;
+  }
+
+  function resetReadTime() {
+    readElapsed = 0;
+    reading = true;
+    readStart = performance.now();
+    clearTimeout(readInactivityTimer);
+  }
+
+  function sendClosePing() {
+    if (closePingedFor === currentPath) return;
+    const read = Math.max(0, Math.round(takeReadTime() / 1000));
+    if (read <= 0) return;
+    const hide = ssoAvailable && isAdmin ? 1 : 0;
+    const body = JSON.stringify({ fr: currentPath, hide, read });
+    const blob = new Blob([body], { type: "application/json" });
+    try {
+      if (navigator.sendBeacon) {
+        navigator.sendBeacon("/_a", blob);
+      } else {
+        fetch("/_a", {
+          method: "POST",
+          keepalive: true,
+          headers: { "content-type": "application/json" },
+          body,
+        });
+      }
+    } catch { /* analytics must never break navigation */ }
+    closePingedFor = currentPath;
+  }
+
+  for (const ev of ["mousemove", "mousedown", "touchstart", "touchmove", "scroll", "keydown"]) {
+    addEventListener(ev, markReadActivity, { passive: true });
+  }
+  addEventListener("pagehide", sendClosePing);
+
   // The initial page load pings too — it is what starts the visit and
   // counts the entry page view (the document GET alone records nothing).
   // Sent once per load, after the auth probes so the admin gate applies;
@@ -538,7 +616,10 @@ import "overlayscrollbars/overlayscrollbars.css";
     if (url.origin !== location.origin) {
       // External link: the browser navigates; record the full https URL so
       // different links to the same domain stay distinct in analytics.
-      if (url.protocol === "https:") ping(url.href);
+      if (url.protocol === "https:") {
+        closePingedFor = currentPath;
+        ping(url.href, currentPath, takeReadTime());
+      }
       return;
     }
     // Same-page anchor links (footnotes etc.): let the browser handle them
@@ -550,7 +631,12 @@ import "overlayscrollbars/overlayscrollbars.css";
     ev.preventDefault();
     // Capture the source now: load() updates currentPath before pinging.
     const from = currentPath;
-    load(url).then((ok) => { if (ok) ping(url.pathname, from); });
+    load(url).then((ok) => {
+      if (!ok) return;
+      closePingedFor = null;
+      ping(url.pathname, from, takeReadTime());
+      resetReadTime();
+    });
   });
 
   addEventListener("popstate", () => {

pagerite/analytics.py

@@ -5,15 +5,21 @@ ping on page load starts a visit, later pings extend it, and pings with no
 known session start a fresh one (missing data, not dropped). The document
 GET handler stashes the entry referer (external https origin) and any
 utm_* query parameters in in-memory IP tables, consumed when the ping
-starts the visit; nothing is counted without a ping (bots and admin
-browsing stay invisible). The session map is in-memory only.  The visitor
-IP and, when available, its reverse-DNS host name are stored on the visit
-record itself.
+starts the visit; nothing is counted without a ping (bots stay invisible).
+Admin clients ping with ``hide=1``, which records nothing and removes any
+visit the session accumulated before logging in.  Scanner telltale 404s
+(dotpaths, *.php) classify the source IP as abuse; its hits — including
+earlier crawler hits — are moved to the abuse list, which the viewer
+groups by IP with full request paths.  Client metadata (IP, UA, language,
+country/city, host) is stored once per unique client hash and referenced
+from visits, crawler hits and abuse hits.  The session map is in-memory
+only.
 
 Data is a msgspec Struct JSON-dumped to its own file (not the kanta db),
 rewritten atomically on every recorded event.
 """
 
+import ipaddress
 import os
 import re
 import tempfile
@@ -23,6 +29,7 @@ from datetime import UTC, datetime, timedelta
 from pathlib import Path
 from urllib.parse import parse_qs, urlparse
 
+import blake3
 import msgspec
 from ua_parser import parse
 
@@ -41,7 +48,10 @@ def _compact_user_agent(ua: str) -> str:
     dev = r.device.family if r.device else None
     if browser in (None, "Other") and os_name in (None, "Other"):
         return ua
-    browser = browser if browser and browser != "Other" else ""
+    if browser and browser != "Other":
+        browser = browser.split()[0]
+    else:
+        browser = ""
     os_name = os_name if os_name and os_name != "Other" else ""
     if dev in (None, "Other") or dev == browser:
         dev = ""
@@ -49,53 +59,90 @@ def _compact_user_agent(ua: str) -> str:
     return " ".join(p for p in parts if p).strip()
 
 
+class Client(msgspec.Struct, omit_defaults=True):
+    """Client metadata shared by visits, crawler hits and abuse hits.
+
+    Identified by a 6-byte blake3 hash of the IPv4 address or IPv6 /64
+    network, the full User-Agent string and the extracted language tag.
+    Country/city/host are filled in asynchronously after the first event.
+    """
+
+    #: Visitor IP address (first X-Forwarded-For hop or direct peer).
+    ip: str = ""
+    #: Reverse-DNS host name for ``ip`` when resolvable, else "".
+    host: str = ""
+    #: First Accept-Language tag, lowercased (e.g. "en-us").
+    lang: str = ""
+    #: Two-letter country code from the DB-IP geoip lookup, or "".
+    country: str = ""
+    #: City name from the DB-IP geoip lookup, or "".
+    city: str = ""
+    #: Raw User-Agent header.
+    ua: str = ""
+    #: Compact display form of ``ua`` (browser/OS/device) when parsable.
+    ua_pretty: str = ""
+
+
 class Visit(msgspec.Struct, omit_defaults=True):
     """One visit: the initial-load data plus everything seen afterwards.
 
     ``trail`` holds page paths and external exit URLs in first-seen
     order; re-visiting an already seen page does not append. The entry
-    page itself is in ``entry``, not in the trail.
+    page itself is in ``entry``, not in the trail.  Client metadata is
+    held in ``Analytics.clients`` keyed by ``client``.
     """
 
     start: datetime
     entry: str
     #: External https origin of the initial load, "" for direct visits.
     referer: str = ""
-    #: Visitor IP address (first X-Forwarded-For hop or direct peer).
-    ip: str = ""
-    #: Reverse-DNS host name for ``ip`` when resolvable, else "".
-    host: str = ""
+    #: 6-byte blake3 hash referencing ``Analytics.clients``.
+    client: bytes = b""
     trail: list[str] = []
-    #: First Accept-Language tag, lowercased (e.g. "en-us").
-    lang: str = ""
-    #: Two-letter region subtag derived from ``lang`` (e.g. "US"), or "".
-    #: Overwritten by the DB-IP geoip lookup when a database is available.
-    country: str = ""
-    #: City name from the DB-IP geoip lookup, or "".
-    city: str = ""
-    #: Raw User-Agent header from the initial ping.
-    ua: str = ""
-    #: Compact display form of ``ua`` (browser/OS/device) when parsable.
-    ua_pretty: str = ""
     #: UTM query parameters from the landing URL, keyed by parameter name.
     utm: dict[str, str] = {}
+    #: Active reading time per path (seconds), keyed by path.
+    read: dict[str, int] = {}
 
 
 class CrawlerHit(msgspec.Struct, omit_defaults=True):
-    """A document GET that was never followed by an analytics ping."""
+    """A document GET that was never followed by an analytics ping.
+
+    Client metadata is held in ``Analytics.clients`` keyed by ``client``.
+    """
 
     start: datetime
     entry: str
-    ip: str = ""
-    ua: str = ""
-    #: Compact display form of ``ua`` when parsable.
-    ua_pretty: str = ""
+    #: 6-byte blake3 hash referencing ``Analytics.clients``.
+    client: bytes = b""
     #: External https origin of the initial load, "" for direct/none.
     referer: str = ""
     #: Raw query string of the landing URL (UTM tags can be parsed from it).
     query: str = ""
 
 
+class AbuseHit(msgspec.Struct, omit_defaults=True):
+    """A request from an IP classified as a scanner/abuser.
+
+    Unlike crawler hits the full request path (query string included) is
+    kept: the interesting part is exactly which paths were probed.
+    ``flag`` marks the path that triggered classification; ``is_404``
+    distinguishes 404 responses from document GETs made by the abuser.
+    Client metadata is held in ``Analytics.clients`` keyed by ``client``.
+    """
+
+    start: datetime
+    #: Full request path including the query string (e.g. "/.env?x=1").
+    path: str
+    #: 6-byte blake3 hash referencing ``Analytics.clients``.
+    client: bytes = b""
+    #: True when this path triggered abuse classification (telltale path
+    #: or the 404 that crossed the threshold).
+    flag: bool = False
+    #: True for 404 responses; false for document GETs from the abuser.
+    is_404: bool = False
+
+
 class Analytics(msgspec.Struct, omit_defaults=True):
     """Root of the analytics JSON file. Append-only by design: old data is
     dropped by deleting list entries / bucket keys."""
@@ -103,6 +150,12 @@ class Analytics(msgspec.Struct, omit_defaults=True):
     visits: list[Visit] = []
     #: Document GETs that never produced a ping, treated as crawler/bot hits.
     crawlers: list[CrawlerHit] = []
+    #: Requests from abusive IPs (see AbuseHit), grouped by IP in the viewer.
+    abuse: list[AbuseHit] = []
+    #: Client metadata keyed by 6-byte blake3 hash.
+    clients: dict[bytes, Client] = {}
+    #: IPs classified as scanners/abusers (keys; values always True).
+    abuse_ips: dict[str, bool] = {}
     #: Page transitions per 5-minute bucket (sparse):
     #: from -> to -> bucket ISO -> count. ``from`` is the referer origin or
     #: "(direct)" for initial loads, a page path for pings.
@@ -186,9 +239,50 @@ def _utm_tags(query: str) -> dict[str, str]:
 
 _CRAWLER_TIMEOUT = timedelta(seconds=10)
 
+#: Plain-404 count per IP that classifies it as abuse even without a
+#: telltale path hit.
+_ABUSE_404_THRESHOLD = 10
+
+#: Paths that instantly classify an IP as abuse when they 404: any segment
+#: starting with a dot ("/.env", "/.git/config") or ending in ".php".
+_ABUSE_PATH = re.compile(r"(^|/)\.|\.php$", re.IGNORECASE)
+
+
+def _is_abuse_path(path: str) -> bool:
+    """Telltale scanner path: dot segment or *.php."""
+    return bool(_ABUSE_PATH.search(path.split("?")[0]))
+
+
+def _network_ip(ip: str) -> str:
+    """IPv4 address unchanged, IPv6 collapsed to its /64 network address.
+
+    We hash the network rather than the full address so that clients in the
+    same /64 (a typical end-user allocation) are treated as one visitor.
+    """
+    if not ip:
+        return ip
+    try:
+        addr = ipaddress.ip_address(ip)
+    except ValueError:
+        return ip
+    if isinstance(addr, ipaddress.IPv6Address):
+        return str(ipaddress.IPv6Network(f"{ip}/64", strict=False).network_address)
+    return ip
+
+
+def _client_hash(ip: str, ua: str, lang: str) -> bytes:
+    """6-byte blake3 digest identifying a visitor/client tuple.
+
+    The key is the prettified IP (IPv6 /64), the raw UA string and the
+    extracted language tag, separated by null bytes.
+    """
+    return blake3.blake3(
+        f"{_network_ip(ip)}\0{ua}\0{lang}".encode()
+    ).digest()[:6]
+
 
 class Store:
-    """In-memory analytics data plus the (IP, UA) -> visit session map."""
+    """In-memory analytics data plus the client-hash -> visit session map."""
 
     def __init__(self, path: Path) -> None:
         self.path = path
@@ -198,8 +292,8 @@ class Store:
                 self.data = msgspec.json.decode(path.read_bytes(), type=Analytics)
             except msgspec.DecodeError, OSError:
                 pass  # legacy schema / corrupt or unreadable file: start fresh
-        #: (ip, user-agent) -> index of the current visit in data.visits
-        self.sessions: dict[tuple[str, str], int] = {}
+        #: client hash -> index of the current visit in data.visits
+        self.sessions: dict[bytes, int] = {}
         #: ip -> external https origin of the latest document GET carrying
         #: one, stashed for the visit the client's initial ping starts.
         #: Internal or absent referers never touch the table.
@@ -212,6 +306,9 @@ class Store:
         #: Document GETs that have not yet been matched by a ping.  Kept
         #: in RAM only; expired entries are written to ``data.crawlers``.
         self.pending_crawlers: list[CrawlerHit] = []
+        #: ip -> number of plain (non-telltale) 404s seen, in RAM only;
+        #: reaching ``_ABUSE_404_THRESHOLD`` classifies the IP as abuse.
+        self.not_found_counts: dict[str, int] = {}
         #: Callables to notify when persisted data changes.  Registered by the
         #: analytics WebSocket broadcaster.
         self._on_change: list[Callable[[], None]] = []
@@ -244,20 +341,26 @@ class Store:
         else:
             self._notify()
 
-    def _flush_crawlers(self, now: datetime | None = None) -> None:
-        """Move expired pending crawler hits into persistent ``data.crawlers``."""
+    def _flush_crawlers(self, now: datetime | None = None) -> list[bytes]:
+        """Move expired pending crawler hits into persistent ``data.crawlers``.
+
+        Returns the client hashes of the newly flushed hits so callers can
+        schedule async enrichment.
+        """
         if not self.pending_crawlers:
-            return
+            return []
         now = now or datetime.now(UTC)
         cutoff = now - _CRAWLER_TIMEOUT
         expired: list[CrawlerHit] = []
         remaining: list[CrawlerHit] = []
         for hit in self.pending_crawlers:
             (expired if hit.start <= cutoff else remaining).append(hit)
-        if expired:
-            self.pending_crawlers = remaining
-            self.data.crawlers.extend(expired)
-            self._save()
+        if not expired:
+            return []
+        self.pending_crawlers = remaining
+        self.data.crawlers.extend(expired)
+        self._save()
+        return [hit.client for hit in expired]
 
     def _count(self, table: dict[str, int], key: str) -> None:
         table[key] = table.get(key, 0) + 1
@@ -267,15 +370,188 @@ class Store:
         buckets = self.data.transitions.setdefault(fr, {}).setdefault(to, {})
         self._count(buckets, _bucket(now))
 
+    def _uncount(self, table: dict[str, int], key: str) -> None:
+        """Reverse one ``_count``: decrement and drop empty keys."""
+        if key in table:
+            table[key] -= 1
+            if table[key] <= 0:
+                del table[key]
+
+    def _remove_visit(self, index: int) -> None:
+        """Delete a visit and reverse the counts its creation recorded.
+
+        Used when a known visitor turns out to be an admin (hide=1 ping):
+        the session is scrubbed from the stats.  Views/transitions logged
+        by later pings inside the visit lack per-event timestamps and are
+        left as-is.
+        """
+        visit = self.data.visits[index]
+        bucket = _bucket(visit.start)
+        self._uncount(self.data.site_visits, bucket)
+        views = self.data.views.get(visit.entry)
+        if views is not None:
+            self._uncount(views, bucket)
+            if not views:
+                del self.data.views[visit.entry]
+        fr_map = self.data.transitions.get(visit.referer or "(direct)")
+        if fr_map is not None:
+            buckets = fr_map.get(visit.entry)
+            if buckets is not None:
+                self._uncount(buckets, bucket)
+                if not buckets:
+                    del fr_map[visit.entry]
+            if not fr_map:
+                del self.data.transitions[visit.referer or "(direct)"]
+        del self.data.visits[index]
+        # Sessions store list indices; shift the ones past the removed visit.
+        for key, i in list(self.sessions.items()):
+            if i > index:
+                self.sessions[key] = i - 1
+
+    def _client_ip(self, client_hash: bytes) -> str:
+        """Return the IP stored for ``client_hash``, or "" if missing."""
+        client = self.data.clients.get(client_hash)
+        return client.ip if client else ""
+
+    def _ensure_client(
+        self,
+        ip: str,
+        ua: str,
+        lang: str,
+        *,
+        country: str = "",
+    ) -> bytes:
+        """Get or create a ``Client`` record; return its 6-byte hash."""
+        h = _client_hash(ip, ua, lang)
+        if h not in self.data.clients:
+            self.data.clients[h] = Client(
+                ip=ip,
+                ua=ua,
+                ua_pretty=_compact_user_agent(ua),
+                lang=lang,
+                country=country,
+            )
+            self._save()
+        return h
+
+    def enrich_client(
+        self,
+        client_hash: bytes,
+        *,
+        host: str = "",
+        country: str = "",
+        city: str = "",
+    ) -> None:
+        """Fill in host/geoip fields on a client record after async lookups."""
+        client = self.data.clients.get(client_hash)
+        if client is None:
+            return
+        changed = False
+        if host and not client.host:
+            client.host = host
+            changed = True
+        if country:
+            client.country = country
+            changed = True
+        if city:
+            client.city = city
+            changed = True
+        if changed:
+            self._save()
+
+    def _abuse_hit(
+        self,
+        client_hash: bytes,
+        path: str,
+        start: datetime | None = None,
+        *,
+        flag: bool = False,
+        is_404: bool = False,
+    ) -> None:
+        """Append one abuse hit referencing a client by hash."""
+        self.data.abuse.append(
+            AbuseHit(
+                start=start or datetime.now(UTC),
+                path=path,
+                client=client_hash,
+                flag=flag,
+                is_404=is_404,
+            )
+        )
+
+    def classify_abuse(
+        self,
+        ip: str,
+        client_hash: bytes,
+        path: str,
+        *,
+        flag: bool = False,
+        is_404: bool = False,
+    ) -> None:
+        """Classify an IP as a scanner/abuser and record the triggering hit.
+
+        All earlier crawler hits from the same IP (persisted and pending)
+        are moved to the abuse list — a random-UA scanner must not pollute
+        the crawler stats of the legitimate bots it impersonates.
+        """
+        if ip not in self.data.abuse_ips:
+            self.data.abuse_ips[ip] = True
+            moved = [h for h in self.data.crawlers if self._client_ip(h.client) == ip]
+            if moved:
+                self.data.crawlers = [h for h in self.data.crawlers if self._client_ip(h.client) != ip]
+                for h in moved:
+                    self._abuse_hit(
+                        h.client,
+                        h.entry + (f"?{h.query}" if h.query else ""),
+                        start=h.start,
+                    )
+            pending = [h for h in self.pending_crawlers if self._client_ip(h.client) == ip]
+            if pending:
+                self.pending_crawlers = [h for h in self.pending_crawlers if self._client_ip(h.client) != ip]
+                for h in pending:
+                    self._abuse_hit(
+                        h.client,
+                        h.entry + (f"?{h.query}" if h.query else ""),
+                        start=h.start,
+                    )
+        self._abuse_hit(client_hash, path, flag=flag, is_404=is_404)
+        self._save()
+
+    def track_404(
+        self,
+        ip: str,
+        ua: str,
+        path: str,
+        accept_language: str = "",
+    ) -> bytes:
+        """Record a 404 response for ``path`` (full path, query included).
+
+        A telltale path (dot segment or *.php) classifies the IP as abuse
+        immediately; enough plain 404s from one IP do too.  Hits from
+        already-classified IPs go straight to the abuse list.
+
+        Returns the client hash so callers can schedule async enrichment.
+        """
+        lang, country = _parse_accept_language(accept_language)
+        client_hash = self._ensure_client(ip, ua, lang, country=country)
+        if ip in self.data.abuse_ips:
+            self._abuse_hit(client_hash, path, flag=_is_abuse_path(path), is_404=True)
+            self._save()
+            return client_hash
+        if _is_abuse_path(path):
+            self.classify_abuse(ip, client_hash, path, flag=True, is_404=True)
+            return client_hash
+        self.not_found_counts[ip] = self.not_found_counts.get(ip, 0) + 1
+        if self.not_found_counts[ip] >= _ABUSE_404_THRESHOLD:
+            self.classify_abuse(ip, client_hash, path, flag=True, is_404=True)
+            return client_hash
+        return client_hash
+
     def _new_visit(
         self,
         entry: str,
         referer: str,
-        key: tuple[str, str],
-        ip: str = "",
-        lang: str = "",
-        country: str = "",
-        ua: str = "",
+        client_hash: bytes,
         utm: dict[str, str] | None = None,
     ) -> Visit:
         now = datetime.now(UTC)
@@ -283,54 +559,25 @@ class Store:
             start=now,
             entry=entry,
             referer=referer,
-            ip=ip,
-            lang=lang,
-            country=country,
-            ua=ua,
-            ua_pretty=_compact_user_agent(ua),
+            client=client_hash,
             utm=utm or {},
         )
         self.data.visits.append(visit)
-        self.sessions[key] = len(self.data.visits) - 1
+        self.sessions[client_hash] = len(self.data.visits) - 1
         self._count(self.data.site_visits, _bucket(now))
         self._count(self.data.views.setdefault(entry, {}), _bucket(now))
         self._count_transition(referer or "(direct)", entry, now)
         return visit
 
-    def enrich_visit(
-        self,
-        index: int,
-        *,
-        host: str = "",
-        country: str = "",
-        city: str = "",
-    ) -> None:
-        """Fill in host/geoip fields on an existing visit after async lookups."""
-        if index < 0 or index >= len(self.data.visits):
-            return
-        visit = self.data.visits[index]
-        changed = False
-        if host and not visit.host:
-            visit.host = host
-            changed = True
-        if country:
-            visit.country = country
-            changed = True
-        if city:
-            visit.city = city
-            changed = True
-        if changed:
-            self._save()
-
     def track_entry(
         self,
         referer: str,
         own_origin: str,
         ip: str,
         ua: str,
-        entry: str,
-        query: str = "",
-    ) -> None:
+        full_path: str,
+        accept_language: str = "",
+    ) -> list[bytes]:
         """Stash the entry referer/UTM tags and queue a pending crawler hit.
 
         Nothing is counted here — the client's initial /_a ping starts the
@@ -341,11 +588,28 @@ class Store:
         does not erase an earlier tagged landing.
 
         Every document GET is also queued as a pending crawler hit.  If a ping
-        from the same (IP, UA) pair arrives within ``_CRAWLER_TIMEOUT``, the
-        hit is discarded; otherwise it is flushed to ``data.crawlers``.
+        from the same client arrives within ``_CRAWLER_TIMEOUT``, the hit is
+        discarded; otherwise it is flushed to ``data.crawlers``.  The
+        Accept-Language header is stored on the client record immediately;
+        host/geoip are filled in later by async enrichment.
+
+        GETs from IPs already classified as abuse are recorded as abuse hits
+        with the full request path (query string included).
+
+        Returns the client hashes of any hits flushed to persistent storage,
+        so callers can schedule async enrichment.
         """
+        entry = full_path.split("?")[0]
+        query = full_path.split("?", 1)[1] if "?" in full_path else ""
+        lang, country = _parse_accept_language(accept_language)
+        client_hash = self._ensure_client(ip, ua, lang, country=country)
+        if ip in self.data.abuse_ips:
+            flushed = self._flush_crawlers()
+            self._abuse_hit(client_hash, full_path, is_404=False, flag=False)
+            self._save()
+            return flushed
         now = datetime.now(UTC)
-        self._flush_crawlers(now)
+        flushed = self._flush_crawlers(now)
         if referer:
             origin = _origin(referer)
             if origin is not None and origin != own_origin:
@@ -357,61 +621,98 @@ class Store:
             CrawlerHit(
                 start=now,
                 entry=entry,
-                ip=ip,
-                ua=ua,
-                ua_pretty=_compact_user_agent(ua),
+                client=client_hash,
                 referer=self.pending_referers.get(ip, ""),
                 query=query,
             )
         )
+        return flushed
+
+    def _add_read(self, client_hash: bytes, path: str, seconds: int) -> None:
+        """Add ``seconds`` of reading time for ``path`` to the current visit."""
+        if seconds <= 0:
+            return
+        index = self.sessions.get(client_hash)
+        if index is None or index >= len(self.data.visits):
+            return
+        visit = self.data.visits[index]
+        visit.read[path] = visit.read.get(path, 0) + seconds
 
     def ping(
         self,
         from_: str,
-        to: str,
+        to: str | None,
         ip: str,
         ua: str,
         accept_language: str = "",
-    ) -> int | None:
-        """Record a client navigation ping ({from, to} from pagerite.js).
+        hide: bool = False,
+        read: int = 0,
+    ) -> tuple[int | None, list[bytes]]:
+        """Record a client navigation ping ({from, to, read} from pagerite.js).
+
+        ``to`` is an internal path ("/...") or an https URL for exit links; a
+        missing/empty ``to`` means the page is being closed and only the
+        ``read`` time should be recorded. The transition is always counted when
+        ``to`` is present; the trail only grows on first sight of a page within
+        the visit. ``read`` is the active time (seconds) spent on ``from_``.
 
-        ``to`` is an internal path ("/...") or an https URL for exit links;
-        anything else is ignored. The transition is always counted; the trail
-        only grows on first sight of a page within the visit.
         A ping with no known session starts a fresh visit, consuming the
         referer and UTM tags stashed by the document GET if there are any.
 
-        Returns the index of the new visit when one is created, so callers
-        can enrich it later with non-blocking lookups (host, geoip country).
+        ``hide`` is set by admin clients: the ping cancels pending crawler
+        hits as usual, and any existing visit for this client session is
+        removed from the stats (the admin browsed anonymously before logging
+        in).  Nothing new is recorded.
+
+        Pings from IPs classified as abuse are ignored entirely.
+
+        Returns the index of the new visit when one is created (or None) and
+        the client hashes of any crawler hits flushed by this call, so callers
+        can schedule async enrichment (host, geoip country/city).
         """
-        self._flush_crawlers()
-        # A real visitor ping cancels any pending crawler hits from this
-        # (IP, UA) pair.
+        flushed = self._flush_crawlers()
+        lang, country = _parse_accept_language(accept_language)
+        client_hash = _client_hash(ip, ua, lang)
+        if hide:
+            # Admin ping: cancel pending crawler hits and scrub the session.
+            self.pending_crawlers = [
+                hit for hit in self.pending_crawlers if hit.client == client_hash
+            ]
+            index = self.sessions.pop(client_hash, None)
+            if index is not None and index < len(self.data.visits):
+                self._remove_visit(index)
+                self._save()
+            return None, flushed
+        if ip in self.data.abuse_ips:
+            return None, flushed
+        # A real visitor ping cancels any pending crawler hits from this client.
         self.pending_crawlers = [
-            hit for hit in self.pending_crawlers if not (hit.ip == ip and hit.ua == ua)
+            hit for hit in self.pending_crawlers if hit.client != client_hash
         ]
+        fr_path = _internal_path(from_) if from_ else ""
+        if fr_path and read > 0:
+            self._add_read(client_hash, fr_path, read)
+        if not to:
+            if read > 0:
+                self._save()
+            return None, flushed
         if to.startswith("/") and not to.startswith("//"):
             target = _internal_path(to) or ""
         else:
             target = _external_target(to) or ""
         if not target:
-            return None
-        key = (ip, ua)
-        index = self.sessions.get(key)
-        fr = (_internal_path(from_) or "(direct)") if from_ else "(direct)"
+            return None, flushed
+        index = self.sessions.get(client_hash)
+        fr = fr_path or "(direct)"
         if index is None or index >= len(self.data.visits):
             # No known session: the initial ping of a fresh page load (or
             # missing data after a server restart) — start a visit.
-            lang, country = _parse_accept_language(accept_language)
             index = len(self.data.visits)
+            self._ensure_client(ip, ua, lang, country=country)
             self._new_visit(
                 target,
                 self.pending_referers.pop(ip, ""),
-                key,
-                ip=ip,
-                lang=lang,
-                country=country,
-                ua=ua,
+                client_hash,
                 utm=self.pending_utms.pop(ip, {}),
             )
         else:
@@ -424,4 +725,5 @@ class Store:
             if visit.entry != target and target not in visit.trail:
                 visit.trail.append(target)
         self._save()
-        return index if index is not None and index < len(self.data.visits) else None
+        visit_index = index if index is not None and index < len(self.data.visits) else None
+        return visit_index, flushed

pagerite/app.py

@@ -126,13 +126,21 @@ class GeoIP:
         return ""
 
     def city(self, ip: str) -> str:
-        """City name for ``ip``, or "" when unavailable."""
+        """City name for ``ip``, or "" when unavailable.
+
+        GeoIP sometimes appends district names in parentheses (e.g.
+        "Berlin (Bezirk Tempelhof-Schöneberg)"); those are stripped before
+        the value is stored.
+        """
         if not ip or self._reader is None:
             return ""
         try:
             rec = self._reader.get(ip)
             if rec:
-                return (rec.get("city") or {}).get("names", {}).get("en", "")
+                city = (rec.get("city") or {}).get("names", {}).get("en", "")
+                if city:
+                    city = re.sub(r"\s*\([^)]*\)", "", city).strip()
+                return city
         except Exception:
             pass
         return ""
@@ -605,6 +613,12 @@ def _client_ip(request: Request) -> str:
     return forwarded or (request.client.host if request.client else "")
 
 
+def _query_suffix(request: Request) -> str:
+    """The request's query string as a "?..." suffix, or "" when absent."""
+    query = str(request.url.query)
+    return f"?{query}" if query else ""
+
+
 @lru_cache(maxsize=4096)
 def _cached_ptr(ip: str) -> str:
     """Reverse-DNS lookup with in-RAM LRU cache.  Returns the host name or ""."""
@@ -638,14 +652,21 @@ async def _geoip_city(ip: str) -> str:
     return await asyncio.to_thread(_geoip.city, ip)
 
 
-async def _enrich_visit(index: int, ip: str) -> None:
-    """Run non-blocking reverse-DNS and geoip enrichment for a new visit."""
-    if not ip:
+async def _enrich_client(client_hash: bytes) -> None:
+    """Run non-blocking reverse-DNS and geoip enrichment for a client."""
+    client = analytics_store.data.clients.get(client_hash)
+    if not client or not client.ip:
         return
-    host = await _lookup_host(ip)
-    country = await _geoip_country(ip)
-    city = await _geoip_city(ip)
-    analytics_store.enrich_visit(index, host=host, country=country, city=city)
+    host = await _lookup_host(client.ip)
+    country = await _geoip_country(client.ip)
+    city = await _geoip_city(client.ip)
+    analytics_store.enrich_client(client_hash, host=host, country=country, city=city)
+
+
+def _schedule_client_enrichment(client_hashes: list[bytes]) -> None:
+    """Start background host/geoip enrichment for the given client hashes."""
+    for client_hash in client_hashes:
+        asyncio.create_task(_enrich_client(client_hash))
 
 
 async def _broadcast_analytics() -> None:
@@ -683,7 +704,11 @@ class AnalyticsPing(BaseModel):
     """Navigation ping from pagerite.js (see docs/analytics.md)."""
 
     fr: str = ""
-    to: str
+    to: str | None = None
+    #: 1 from admin clients: scrub the session instead of recording it.
+    hide: int = 0
+    #: Active reading time on ``fr`` (ms), if any.
+    read: int = 0
 
 
 @app.get("/_a", response_model=None)
@@ -710,42 +735,51 @@ async def analytics_ping(ping: AnalyticsPing, request: Request) -> None:
     the response is never delayed by slow DNS or the first MMDB decompress.
     """
     ip = _client_ip(request)
-    index = analytics_store.ping(
+    visit_index, flushed_clients = analytics_store.ping(
         ping.fr,
         ping.to,
         ip,
         request.headers.get("user-agent", ""),
         request.headers.get("accept-language", ""),
+        hide=bool(ping.hide),
+        read=ping.read,
     )
-    if index is not None:
-        asyncio.create_task(_enrich_visit(index, ip))
+    if visit_index is not None:
+        visit = analytics_store.data.visits[visit_index]
+        asyncio.create_task(_enrich_client(visit.client))
+    _schedule_client_enrichment(flushed_clients)
 
 
-def _track_entry(path: str, request: Request) -> None:
+def _track_entry(path: str, request: Request) -> list[bytes]:
     """Stash the referer/UTM tags and queue a pending crawler hit for the GET.
 
     Nothing is counted on the GET itself — the client's /_a ping starts the
-    visit, so bots and admin browsing never register as visits.
+    visit, so bots never register as visits.  (Admin clients ping too, but
+    with hide=1, which scrubs their session instead of recording it.)
 
     The devserver's health probe (``GET /?from=devserver.py`` from
     ``127.0.0.1``) is ignored: it is not real traffic and would otherwise be
     logged as a crawler hit.  The root-path and localhost checks prevent
     remote visitors from hiding traffic with the same query string.
+
+    Returns the client hashes of any pending crawler hits flushed to persistent
+    storage, so callers can schedule async geoip and reverse-DNS enrichment.
     """
     if (
         path == ""
         and str(request.url.query) == "from=devserver.py"
         and _client_ip(request) == "127.0.0.1"
     ):
-        return
+        return []
     own_origin = f"https://{urlparse(str(request.base_url)).netloc}"
-    analytics_store.track_entry(
+    full_path = f"{request.url.path}{_query_suffix(request)}"
+    return analytics_store.track_entry(
         request.headers.get("referer", ""),
         own_origin,
         _client_ip(request),
         request.headers.get("user-agent", ""),
-        "/" if path == "" else f"/{path}",
-        str(request.url.query),
+        full_path,
+        request.headers.get("accept-language", ""),
     )
 
 
@@ -965,9 +999,20 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
     placeholder page (nav links point straight at its first child).
     """
     path = path.strip("/")
+    ua = request.headers.get("user-agent", "")
+    accept_language = request.headers.get("accept-language", "")
     if path and _is_reserved(path):
         # Invalid slug shape: not a content URL, let FastAPI return its
         # built-in 404 instead of rendering an editable article page.
+        # Scanner telltales (dotpaths like /.env, *.php) classify the IP
+        # as abuse in analytics.
+        client_hash = analytics_store.track_404(
+            _client_ip(request),
+            ua,
+            f"/{path}{_query_suffix(request)}",
+            accept_language,
+        )
+        asyncio.create_task(_enrich_client(client_hash))
         raise HTTPException(404)
     chain = resolve(data.menu, path)
     node = chain[-1] if chain else None
@@ -982,7 +1027,8 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
         if request.headers.get("if-none-match") == etag:
             return Response(status_code=304)
         if _is_trackable_path(path):
-            _track_entry(path, request)
+            flushed = _track_entry(path, request)
+            _schedule_client_enrichment(flushed)
         return HTMLResponse(
             views.render_page(data.menu, path, data.brand, data.custom_css, data.theme, data.favicon, data.brand_html, str(request.base_url).rstrip("/")),
             headers={
@@ -995,7 +1041,8 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
         # Category label without a landing page: placeholder with the pen
         # to create it (404 — no page here, but the node is real).
         if _is_trackable_path(path):
-            _track_entry(path, request)
+            flushed = _track_entry(path, request)
+            _schedule_client_enrichment(flushed)
         return HTMLResponse(
             views.render_category(data.menu, path, data.brand, data.custom_css, data.theme, data.favicon, data.brand_html),
             404,
@@ -1011,5 +1058,13 @@ async def show_page(request: Request, path: str) -> HTMLResponse | Response:
             if item.published:
                 return RedirectResponse(f"/{slug}")
     if _is_trackable_path(path):
-        _track_entry(path, request)
+        client_hash = analytics_store.track_404(
+            _client_ip(request),
+            ua,
+            f"/{path}{_query_suffix(request)}",
+            accept_language,
+        )
+        asyncio.create_task(_enrich_client(client_hash))
+        flushed = _track_entry(path, request)
+        _schedule_client_enrichment(flushed)
     return HTMLResponse(views.render_not_found(data.menu, path, data.brand, data.custom_css, data.theme, data.favicon, data.brand_html), 404)

scripts/fake_traffic.py

@@ -6,23 +6,18 @@
 #     "playwright>=1.45.0",
 # ]
 # ///
-"""Generate fake browser visits and crawler hits for a Pagerite site.
+"""Generate fake browser visits, crawler hits, and abuse scans for a Pagerite site.
 
-The script drives a real Chromium browser with Playwright, clicking visible
-internal links so the site's own analytics JavaScript records normal visits
-(POST /_a).  Most browser sessions enter the site with a cross-origin
-``Referer: https://somedomain.com/`` header, and outbound links found on the
-page are followed to real external sites (ending the session).  Browser
-sessions and crawler GETs send a small rotating pool of real public IPs in
-X-Forwarded-For, so the backend can reverse-DNS and GeoIP them instead of seeing
-every hit as 127.0.0.1.
-
-Sessions start with a Poisson inter-arrival delay (``--arrival-rate``) to
-spread traffic out a little, while still keeping the overall run fast.
+Browser sessions (ordinary users) come from realistic residential IPv4 and IPv6
+addresses and stay mostly stable; an IPv6 host part may rotate once mid-session,
+and an IPv4 session may switch to another residential address.  Crawler hits come
+from datacenter IPs, with each crawler profile paired to a matching provider IP
+when possible.  Abuse scanners fire bursts of vulnerability probes from pinned
+datacenter IPs.
 
 Run against a local dev server, e.g.:
 
-    uv run scripts/fake_traffic.py http://localhost:3200 -b 8 -c 20
+    uv run scripts/fake_traffic.py http://localhost:3200
 
 Repeat whenever you want more traffic; each run appends new events to the
 site's analytics file.
@@ -39,7 +34,7 @@ from collections.abc import Sequence
 from dataclasses import dataclass
 from datetime import UTC, datetime
 from typing import Any
-from urllib.parse import urljoin, urlparse
+from urllib.parse import urlencode, urljoin, urlparse
 
 import httpx
 
@@ -59,6 +54,7 @@ class BrowserProfile:
 class CrawlerProfile:
     name: str
     user_agent: str
+    ip: str
 
 
 BROWSER_PROFILES: list[BrowserProfile] = [
@@ -95,39 +91,397 @@ CRAWLER_PROFILES: list[CrawlerProfile] = [
     CrawlerProfile(
         "googlebot",
         "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/128.0.0.0 Safari/537.36",
+        "66.249.64.66",                         # US, Google
     ),
     CrawlerProfile(
         "bingbot",
         "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/128.0.0.0 Safari/537.36",
+        "40.77.167.0",                          # US, Microsoft
     ),
     CrawlerProfile(
-        "duckduckbot", "DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)"
+        "duckduckbot",
+        "DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)",
+        "95.217.0.1",                           # Germany, Hetzner VPS
+    ),
+    CrawlerProfile(
+        "curl",
+        "curl/8.5.0",
+        "139.162.0.1",                          # Singapore, Linode VPS
     ),
-    CrawlerProfile("curl", "curl/8.5.0"),
 ]
 
-# Small pool of real public resolver IPs. They have real reverse-DNS and GeoIP
-# entries, and cycling through a handful avoids hammering DNS during traffic
-# generation.
-SOURCE_IPS: list[str] = [
-    "8.8.8.8",
-    "1.1.1.1",
-    "9.9.9.9",
-    "208.67.222.222",
-    "185.228.168.9",
-    "94.140.14.14",
+# Residential IPv4 addresses and IPv6 /64 prefixes used for ordinary browser
+# sessions.  IPv6 entries keep the network part stable and randomise only the
+# host part; the host may rotate once mid-session.
+RESIDENTIAL_SOURCE_IPS: list[str] = [
+    # Residential IPv4
+    "91.154.140.209",                       # Finland, Elisa
+    "84.143.145.207",                       # Germany, Deutsche Telekom
+    "220.165.255.254",                      # China, Chinanet / China Telecom
+    "84.235.83.162",                        # Saudi Arabia, SaudiNet / STC
+    # Residential IPv6 /64 prefixes
+    "2a02:8109:ac82:6f0c::/64",             # Germany, Deutsche Telekom
+    "240e:45d:1e60:5b0::/64",               # China, China Telecom
+    "2409:8904:6720:4123::/64",             # China, China Unicom
 ]
 
+# Concrete datacenter IPs used for abuse scanner bursts.  They stay pinned for
+# the whole scan burst.
+# Index 0 randomises its UA per request, index 1 uses a fixed browser UA,
+# and index 2 uses a fixed crawler UA.
+ABUSE_SOURCE_IPS: list[str] = [
+    "45.63.0.12",                           # US, Vultr VPS
+    "138.197.0.89",                         # US, DigitalOcean / Cloudways
+    "2a01:4f8:0:2::1234",                   # Germany, Hetzner VPS
+]
+
+# Paths commonly probed by attackers looking for exposed config, admin panels,
+# version control, credentials, backups, or debug endpoints.
+SUSPICIOUS_PATHS: list[str] = [
+    "/.env",
+    "/env",
+    "/.env.local",
+    "/env.development",
+    "/config",
+    "/config.json",
+    "/config.yaml",
+    "/config.yml",
+    "/configuration.json",
+    "/configuration.yaml",
+    "/configuration.yml",
+    "/settings.json",
+    "/settings.yaml",
+    "/settings.yml",
+    "/app.config",
+    "/appsettings.json",
+    "/appsettings.Development.json",
+    "/credentials",
+    "/credentials.json",
+    "/secrets",
+    "/secrets.json",
+    "/.aws/credentials",
+    "/.ssh/id_rsa",
+    "/id_rsa",
+    "/id_rsa.pub",
+    "/known_hosts",
+    "/sftp-config.json",
+    "/admin",
+    "/administrator",
+    "/adminer.php",
+    "/login",
+    "/signin",
+    "/auth/login",
+    "/api/login",
+    "/api/.env",
+    "/api/config",
+    "/api/v1/config",
+    "/api/v2/config",
+    "/webhook",
+    "/webhooks",
+    "/callback",
+    "/proxy",
+    "/image",
+    "/images",
+    "/preview",
+    "/download",
+    "/downloads",
+    "/log",
+    "/logs",
+    "/debug",
+    "/trace",
+    "/phpinfo.php",
+    "/info.php",
+    "/phpmyadmin",
+    "/pma",
+    "/myadmin",
+    "/phpMyAdmin",
+    "/wp-admin",
+    "/wp-login.php",
+    "/wp-config.php",
+    "/xmlrpc.php",
+    "/wp-json/wp/v2/users",
+    "/.git/config",
+    "/.git/HEAD",
+    "/git/config",
+    "/swagger-ui.html",
+    "/v2/api-docs",
+    "/actuator/env",
+    "/actuator/health",
+    "/actuator/configprops",
+    "/server-status",
+    "/.htaccess",
+    "/web.config",
+    "/package.json",
+    "/composer.json",
+    "/vendor/autoload.php",
+    "/docker-compose.yml",
+    "/Dockerfile",
+    "/manage",
+    "/console",
+    "/manager",
+    "/manager/html",
+    "/metrics",
+    "/prometheus",
+    "/healthz",
+    "/_api",
+    "/api",
+    "/api/v1/",
+    "/api/v2/",
+    "/graphql",
+    "/query",
+    "/feed",
+    "/rss",
+    "/_debug",
+    "/test",
+    "/testing",
+    "/tmp",
+    "/temp",
+    "/backup",
+    "/backups",
+    "/dump",
+    "/dumps",
+    "/sql",
+    "/db",
+    "/database",
+    "/dump.sql",
+    "/backup.sql",
+    "/db.sql",
+    "/backup.zip",
+    "/backup.tar.gz",
+    "/site.zip",
+    "/site.tar.gz",
+    "/source.zip",
+    "/src.zip",
+    "/upload",
+    "/uploads",
+    "/import",
+    "/export",
+    "/token",
+    "/tokens",
+    "/oauth",
+    "/oauth2",
+    "/openid",
+    "/jwks",
+    "/keys",
+    "/key",
+    "/private",
+    "/public",
+]
+
+# Realistic external referers.  Most sessions arrive with a generic referer;
+# a subset carries matching UTM tags on the landing URL.
+PLAIN_REFERRERS: list[str] = [
+    "https://example.com/",
+    "https://somedomain.com/",
+    "https://another-site.org/",
+    "https://friend-site.net/",
+]
+
+# (referer origin, utm parameter dict) pairs used for tagged traffic.
+TAGGED_REFERRERS: list[tuple[str, dict[str, str]]] = [
+    ("https://chatgpt.com/", {"utm_source": "chatgpt.com"}),
+    ("https://www.google.com/", {"utm_source": "google", "utm_medium": "organic"}),
+    ("https://twitter.com/", {"utm_source": "twitter", "utm_medium": "social"}),
+    ("https://www.linkedin.com/", {"utm_source": "linkedin", "utm_medium": "social"}),
+    ("https://github.com/", {"utm_source": "github", "utm_medium": "referral"}),
+    ("https://news.ycombinator.com/", {"utm_source": "hackernews", "utm_medium": "referral"}),
+    ("https://www.reddit.com/", {"utm_source": "reddit", "utm_medium": "social"}),
+    ("https://medium.com/", {"utm_source": "medium", "utm_medium": "referral"}),
+    ("https://www.producthunt.com/", {"utm_source": "producthunt", "utm_medium": "referral"}),
+]
+
+# Fraction of referered sessions that also carry UTM tags.
+UTM_RATE = 0.25
+
+# Innocent-looking paths that do not exist on a Pagerite site.  Hitting many of
+# these from a single IP is itself a telltale of a spray-and-pray scanner.
+NORMAL_404_PATHS: list[str] = [
+    "/about",
+    "/about-us",
+    "/services",
+    "/products",
+    "/contact",
+    "/contact-us",
+    "/team",
+    "/careers",
+    "/jobs",
+    "/pricing",
+    "/features",
+    "/demo",
+    "/trial",
+    "/docs",
+    "/documentation",
+    "/api-docs",
+    "/support",
+    "/help",
+    "/faq",
+    "/knowledge-base",
+    "/terms",
+    "/terms-of-service",
+    "/privacy",
+    "/privacy-policy",
+    "/legal",
+    "/blog",
+    "/news",
+    "/articles",
+    "/press",
+    "/events",
+    "/webinars",
+    "/podcast",
+    "/videos",
+    "/resources",
+    "/whitepapers",
+    "/case-studies",
+    "/customers",
+    "/clients",
+    "/testimonials",
+    "/reviews",
+    "/partners",
+    "/integrations",
+    "/api-reference",
+    "/developers",
+    "/status",
+    "/security",
+    "/trust",
+    "/compliance",
+    "/gdpr",
+    "/ccpa",
+    "/sitemap",
+    "/archive",
+    "/tags",
+    "/categories",
+    "/search",
+    "/users",
+    "/accounts",
+    "/dashboard",
+    "/profile",
+    "/settings",
+    "/preferences",
+    "/notifications",
+    "/messages",
+    "/inbox",
+    "/calendar",
+    "/reports",
+    "/analytics",
+    "/billing",
+    "/invoice",
+    "/orders",
+    "/cart",
+    "/checkout",
+    "/store",
+    "/shop",
+    "/home",
+    "/main",
+    "/start",
+    "/welcome",
+    "/intro",
+    "/overview",
+    "/summary",
+    "/portfolio",
+    "/projects",
+    "/work",
+    "/solutions",
+]
+
+ABUSE_USER_AGENTS: list[str] = [
+    # Desktop browsers
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
+    "(KHTML, like Gecko) Version/17.5 Safari/605.1.15",
+    "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0",
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0",
+    "Mozilla/5.0 (Linux; Android 14; SM-S918B) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/128.0.0.0 Mobile Safari/537.36",
+    "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 "
+    "(KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1",
+    # Well-known crawlers / bots
+    "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; "
+    "+http://www.google.com/bot.html) Chrome/128.0.0.0 Safari/537.36",
+    "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; "
+    "+http://www.bing.com/bingbot.htm) Chrome/128.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (compatible; DuckDuckBot/1.1; +http://duckduckgo.com/duckduckbot.html)",
+    "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)",
+    "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/128.0.0.0 Mobile Safari/537.36 "
+    "(compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
+    "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)",
+    "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)",
+    "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)",
+    "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)",
+    # Social / service fetchers
+    "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)",
+    "Twitterbot/1.0",
+    "LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)",
+    "Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)",
+    "WhatsApp/2.23.20.0",
+    # Command-line / library clients
+    "curl/8.5.0",
+    "Wget/1.21.4 (linux-gnu)",
+    "python-requests/2.32.3",
+    "Go-http-client/1.1",
+    "Node.js/20.5.1",
+]
+
+
+def _random_ipv6_host(prefix: str) -> str:
+    """Return a concrete address within an IPv6 /64 prefix.
+
+    The host part is generated randomly, mimicking a fresh OS privacy address.
+    The input prefix must end in ``::/64`` (e.g. ``2a02:8109:ac82:6f0c::/64``).
+    """
+    if "/" not in prefix:
+        return prefix
+    base, mask = prefix.split("/")
+    if mask != "64":
+        raise ValueError(f"only /64 IPv6 prefixes are supported, got {prefix!r}")
+    if base.endswith("::"):
+        base = base[:-2]
+    host = ":".join(f"{random.randint(0, 0xffff):04x}" for _ in range(4))
+    return f"{base}:{host}"
+
+
+def _concretize_ip(entry: str) -> str:
+    """Return a concrete IP address; randomise the host part for IPv6 /64 prefixes."""
+    if ":" in entry and "/" in entry:
+        return _random_ipv6_host(entry)
+    return entry
+
+
+class _SessionIP:
+    """Stable IP for a browser session, with one optional mid-session rotation.
+
+    IPv6 prefixes get a fresh random host part; IPv4 addresses are swapped for
+    another address from the residential pool.
+    """
+
+    def __init__(self, entry: str, pool: Sequence[str]):
+        self.entry = entry
+        self.pool = pool
+        self._value = _concretize_ip(entry)
+
+    def current(self) -> str:
+        return self._value
+
+    def rotate(self) -> None:
+        if ":" in self.entry and "/" in self.entry:
+            self._value = _random_ipv6_host(self.entry)
+            return
+        # IPv4: switch to another IPv4 address from the residential pool.
+        for _ in range(20):
+            candidate_entry = random.choice(self.pool)
+            if ":" in candidate_entry and "/" in candidate_entry:
+                continue
+            candidate = _concretize_ip(candidate_entry)
+            if candidate != self._value:
+                self._value = candidate
+                return
+
 
 def _sleep(base: float, jitter: float) -> None:
     time.sleep(max(0.0, base + random.uniform(-jitter, jitter)))
 
 
-def _source_ip(index: int) -> str:
-    """Pick one of the small pool of real public IPs."""
-    return SOURCE_IPS[index % len(SOURCE_IPS)]
-
-
 def _normalize_url(url: str) -> str:
     """Return a usable base URL, adding missing scheme/host/port parts.
 
@@ -232,43 +586,75 @@ def _run_browser_session(
     paths: Sequence[str],
     profile: BrowserProfile,
     session_index: int,
-    max_clicks: int,
-    stay: tuple[float, float],
-    headless: bool,
-    fake_ip: str,
-    referer_rate: float,
-    include_external: bool = True,
+    ip_entry: str,
 ) -> dict[str, Any]:
     from playwright.sync_api import sync_playwright
 
+    MAX_CLICKS = 6
+    STAY = (2.0, 6.0)
+    HEADLESS = True
+    REFERER_RATE = 0.75
+    INCLUDE_EXTERNAL = True
+
+    ip_provider = _SessionIP(ip_entry, RESIDENTIAL_SOURCE_IPS)
+    ips_used: list[str] = [ip_provider.current()]
     trail: list[str] = []
     start_time = datetime.now(UTC)
     try:
         with sync_playwright() as p:
             browser = p.chromium.launch(
-                headless=headless,
+                headless=HEADLESS,
                 args=["--no-sandbox", "--disable-dev-shm-usage"],
             )
             extra_headers = {
-                "X-Forwarded-For": fake_ip,
+                "X-Forwarded-For": ip_provider.current(),
                 "Accept-Language": profile.accept_language,
             }
             # Most sessions arrive from an external origin; some are direct.
-            if random.random() < referer_rate:
-                extra_headers["Referer"] = "https://somedomain.com/"
+            # A subset of referered sessions carries realistic UTM tags on the
+            # landing URL; the referer origin is paired with the UTM source.
+            tagged: dict[str, str] = {}
+            if random.random() < REFERER_RATE:
+                if random.random() < UTM_RATE:
+                    referer, tagged = random.choice(TAGGED_REFERRERS)
+                else:
+                    referer = random.choice(PLAIN_REFERRERS)
+                extra_headers["Referer"] = referer
             context = browser.new_context(
                 user_agent=profile.user_agent,
                 viewport={"width": profile.viewport[0], "height": profile.viewport[1]},
                 extra_http_headers=extra_headers,
             )
             page = context.new_page()
+
+            # Update X-Forwarded-For per request; the value stays stable unless we
+            # explicitly rotate it once mid-session.
+            def _route_handler(route, request):
+                headers = dict(request.headers)
+                headers["X-Forwarded-For"] = ip_provider.current()
+                ips_used.append(headers["X-Forwarded-For"])
+                route.continue_(headers=headers)
+
+            page.route("**/*", _route_handler)
+
+            # Pick one point during the session to emulate an IP rotation.
+            rotate_at = random.randint(0, MAX_CLICKS - 1) if MAX_CLICKS > 0 else -1
+
             entry = random.choice(paths) if paths else "/"
-            page.goto(urljoin(base, entry), wait_until="networkidle")
+            landing = urljoin(base, entry)
+            if tagged:
+                sep = "&" if "?" in landing else "?"
+                landing += sep + urlencode(tagged)
+            page.goto(landing, wait_until="networkidle")
             trail.append(page.url)
 
-            for _ in range(max_clicks):
-                _sleep(random.uniform(*stay) / 2, 0.3)
-                links = _collect_links(page, include_external)
+            for click_idx in range(MAX_CLICKS):
+                _sleep(random.uniform(*STAY) / 2, 0.3)
+                if click_idx == rotate_at:
+                    ip_provider.rotate()
+                    ips_used.append(ip_provider.current())
+                    logger.debug("rotated session IP to %s", ip_provider.current())
+                links = _collect_links(page, INCLUDE_EXTERNAL)
                 visible = [item for item in links if item.get("visible")]
                 if not visible:
                     visible = links
@@ -291,14 +677,15 @@ def _run_browser_session(
                     break
                 page.wait_for_load_state("networkidle")
                 trail.append(page.url)
-                _sleep(random.uniform(*stay), 0.5)
+                _sleep(random.uniform(*STAY), 0.5)
 
             browser.close()
 
         return {
             "profile": profile.name,
             "entry": entry,
-            "ip": fake_ip,
+            "ip": ips_used[0],
+            "ips_seen": len(set(ips_used)),
             "pages": len(trail),
             "trail": [urlparse(u).path or "/" for u in trail],
             "duration": (datetime.now(UTC) - start_time).total_seconds(),
@@ -312,12 +699,10 @@ def _run_crawler_hit(
     base: str,
     paths: Sequence[str],
     profile: CrawlerProfile,
-    profile_index: int,
-    session_index: int,
 ) -> dict[str, Any]:
     path = random.choice(paths) if paths else "/"
     url = urljoin(base, path)
-    fake_ip = _source_ip(session_index)
+    fake_ip = profile.ip
     headers = {
         "User-Agent": profile.user_agent,
         "X-Forwarded-For": fake_ip,
@@ -337,6 +722,78 @@ def _run_crawler_hit(
         return {"profile": profile.name, "path": path, "error": str(exc)}
 
 
+def _abuse_ua() -> str:
+    """Return a randomized, syntactically valid user agent for an abuse scan."""
+    return random.choice(ABUSE_USER_AGENTS)
+
+
+def _run_abuse_scanner(base: str, ip_index: int) -> dict[str, Any]:
+    """Fire a burst of vulnerability probes from a single fake IP.
+
+    Scanner 0 randomises its user agent every request, scanner 1 uses a fixed
+    browser UA, and scanner 2 uses a fixed crawler UA.
+    """
+    ip_entry = ABUSE_SOURCE_IPS[ip_index % len(ABUSE_SOURCE_IPS)]
+    if ":" in ip_entry and "/" in ip_entry:
+        fake_ip = _random_ipv6_host(ip_entry)
+    else:
+        fake_ip = ip_entry
+
+    MIN_HITS = 15
+    MAX_HITS = 25
+    total_hits = random.randint(MIN_HITS, MAX_HITS)
+
+    # Ensure the burst contains both telltales: suspicious paths and more
+    # than ten normal-looking 404 paths.
+    suspicious_count = max(5, total_hits // 3)
+    normal_count = total_hits - suspicious_count
+    if normal_count < 11:
+        normal_count = 11
+        suspicious_count = max(3, total_hits - normal_count)
+
+    paths = random.choices(SUSPICIOUS_PATHS, k=suspicious_count) + random.choices(
+        NORMAL_404_PATHS, k=normal_count
+    )
+    random.shuffle(paths)
+
+    ua_mode = ip_index % 3
+    if ua_mode == 0:
+        get_ua = _abuse_ua
+    elif ua_mode == 1:
+        def get_ua() -> str:
+            return BROWSER_PROFILES[0].user_agent
+    else:
+        def get_ua() -> str:
+            return CRAWLER_PROFILES[0].user_agent
+
+    scan_results: list[dict[str, Any]] = []
+    with httpx.Client(follow_redirects=True, timeout=15.0) as client:
+        for path in paths:
+            headers = {
+                "User-Agent": get_ua(),
+                "X-Forwarded-For": fake_ip,
+                "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+                "Accept-Language": random.choice(
+                    ["en-US,en;q=0.9", "en-GB,en;q=0.8", "en;q=0.7"]
+                ),
+            }
+            try:
+                r = client.get(urljoin(base, path), headers=headers)
+                scan_results.append(
+                    {"path": path, "status": r.status_code, "ua": headers["User-Agent"]}
+                )
+            except Exception as exc:  # noqa: BLE001
+                scan_results.append({"path": path, "error": str(exc)})
+            _sleep(0.15, 0.1)
+
+    return {
+        "scanner": ip_index + 1,
+        "ip": fake_ip,
+        "hits": len(scan_results),
+        "results": scan_results,
+    }
+
+
 def _parse_args(argv: Sequence[str] | None) -> argparse.Namespace:
     parser = argparse.ArgumentParser(
         description="Generate fake traffic for a Pagerite site.",
@@ -351,54 +808,13 @@ def _parse_args(argv: Sequence[str] | None) -> argparse.Namespace:
              "missing scheme defaults to http://.",
     )
     parser.add_argument(
-        "-b",
-        "--browsers",
-        type=int,
-        default=5,
-        help="Number of simulated browser sessions",
-    )
-    parser.add_argument(
-        "-c", "--crawlers", type=int, default=10, help="Number of crawler HTTP GETs"
-    )
-    parser.add_argument(
-        "--max-clicks",
-        type=int,
-        default=6,
-        help="Max internal link clicks per browser session",
-    )
-    parser.add_argument(
-        "--stay",
+        "-t",
+        "--duration",
         type=float,
-        nargs=2,
-        default=[2.0, 6.0],
-        metavar=("MIN", "MAX"),
-        help="Seconds to stay on a page before clicking again",
+        default=60.0,
+        metavar="SECONDS",
+        help="Rough maximum time to generate traffic (0 runs one preset batch)",
     )
-    parser.add_argument(
-        "--headless",
-        action=argparse.BooleanOptionalAction,
-        default=True,
-        help="Run browsers headlessly",
-    )
-    parser.add_argument(
-        "--arrival-rate",
-        type=float,
-        default=1.0,
-        help="Average arrivals per second (Poisson). 0 disables inter-arrival waits",
-    )
-    parser.add_argument(
-        "--referer-rate",
-        type=float,
-        default=0.75,
-        help="Share of browser sessions that arrive with a cross-origin Referer",
-    )
-    parser.add_argument(
-        "--external-links",
-        action=argparse.BooleanOptionalAction,
-        default=True,
-        help="Include real outbound links in random navigation",
-    )
-    parser.add_argument("--seed", type=int, default=None, help="Random seed")
     parser.add_argument("-v", "--verbose", action="store_true", help="Debug logging")
     return parser.parse_args(argv)
 
@@ -414,8 +830,6 @@ def main(argv: Sequence[str] | None = None) -> int:
         logger.error("%s", exc)
         return 2
 
-    random.seed(args.seed)
-
     # Discover content paths from the public page tree if we can.
     paths: list[str] = []
     try:
@@ -428,61 +842,95 @@ def main(argv: Sequence[str] | None = None) -> int:
         paths = ["/"]
 
     logger.info(
-        "Generating fake traffic against %s (%d content paths, %d browsers, %d crawlers)",
+        "Generating fake traffic against %s (%d content paths, duration=%ss)",
         base,
         len(paths),
-        args.browsers,
-        args.crawlers,
+        args.duration,
     )
 
     results: list[dict[str, Any]] = []
+    arrival_rate = 1.0
 
-    for i in range(args.browsers):
-        if i > 0:
-            wait = _poisson_wait(args.arrival_rate)
-            logger.debug("waiting %.2fs before next browser session", wait)
-            time.sleep(wait)
-        profile = random.choice(BROWSER_PROFILES)
-        fake_ip = _source_ip(i)
-        logger.info(
-            "[%d/%d] browser session: %s (ip=%s)",
-            i + 1,
-            args.browsers,
-            profile.name,
-            fake_ip,
-        )
-        result = _run_browser_session(
-            base,
-            paths,
-            profile,
-            i,
-            args.max_clicks,
-            (args.stay[0], args.stay[1]),
-            args.headless,
-            fake_ip,
-            args.referer_rate,
-            args.external_links,
-        )
-        results.append(result)
-        logger.debug("  trail: %s", result.get("trail", []))
+    def _wait() -> None:
+        wait = _poisson_wait(arrival_rate)
+        logger.debug("waiting %.2fs before next session", wait)
+        time.sleep(wait)
 
-    for i in range(args.crawlers):
-        if i > 0:
-            wait = _poisson_wait(args.arrival_rate)
-            logger.debug("waiting %.2fs before next crawler hit", wait)
-            time.sleep(wait)
-        profile_index = i % len(CRAWLER_PROFILES)
-        profile = CRAWLER_PROFILES[profile_index]
-        fake_ip = _source_ip(i)
-        logger.info(
-            "[%d/%d] crawler hit: %s (ip=%s)",
-            i + 1,
-            args.crawlers,
-            profile.name,
-            fake_ip,
-        )
-        result = _run_crawler_hit(base, paths, profile, profile_index, i)
-        results.append(result)
+    if args.duration <= 0:
+        # One preset batch.
+        for i in range(5):
+            if i > 0:
+                _wait()
+            profile = random.choice(BROWSER_PROFILES)
+            ip_entry = random.choice(RESIDENTIAL_SOURCE_IPS)
+            logger.info(
+                "browser session: %s (ip=%s)",
+                profile.name,
+                _concretize_ip(ip_entry),
+            )
+            result = _run_browser_session(base, paths, profile, i, ip_entry)
+            results.append(result)
+            logger.debug("  trail: %s", result.get("trail", []))
+
+        for i in range(10):
+            if i > 0:
+                _wait()
+            profile = random.choice(CRAWLER_PROFILES)
+            logger.info(
+                "crawler hit: %s (ip=%s)",
+                profile.name,
+                profile.ip,
+            )
+            result = _run_crawler_hit(base, paths, profile)
+            results.append(result)
+
+        for i in range(3):
+            if i > 0:
+                _wait()
+            ip_entry = ABUSE_SOURCE_IPS[i % len(ABUSE_SOURCE_IPS)]
+            logger.info("abuse scanner: %s", ip_entry)
+            result = _run_abuse_scanner(base, i)
+            results.append(result)
+            logger.debug(
+                "  hits: %s", [r.get("path") for r in result.get("results", [])]
+            )
+    else:
+        deadline = time.time() + args.duration
+        session_index = 0
+        while time.time() < deadline:
+            if session_index > 0:
+                _wait()
+            phase = session_index % 3
+            if phase == 0:
+                profile = random.choice(BROWSER_PROFILES)
+                ip_entry = random.choice(RESIDENTIAL_SOURCE_IPS)
+                logger.info(
+                    "browser session: %s (ip=%s)",
+                    profile.name,
+                    _concretize_ip(ip_entry),
+                )
+                result = _run_browser_session(
+                    base, paths, profile, session_index, ip_entry
+                )
+                logger.debug("  trail: %s", result.get("trail", []))
+            elif phase == 1:
+                profile = random.choice(CRAWLER_PROFILES)
+                logger.info(
+                    "crawler hit: %s (ip=%s)",
+                    profile.name,
+                    profile.ip,
+                )
+                result = _run_crawler_hit(base, paths, profile)
+            else:
+                ip_entry = ABUSE_SOURCE_IPS[session_index % len(ABUSE_SOURCE_IPS)]
+                logger.info("abuse scanner: %s", ip_entry)
+                result = _run_abuse_scanner(base, session_index // 3)
+                logger.debug(
+                    "  hits: %s",
+                    [r.get("path") for r in result.get("results", [])],
+                )
+            results.append(result)
+            session_index += 1
 
     ok = sum(1 for r in results if "error" not in r)
     logger.info("Done: %d/%d requests succeeded.", ok, len(results))