Record robots.txt/sitemap.xml GETs, emoji-mark machinery steps in trails

The robots and sitemap routes never called _record_get, so those
accesses were invisible. Recorded like page GETs, they surface as
crawler hits at display time (no activity message ever follows). In
the trail rendering, known non-article machinery paths (robots,
sitemap, future feed paths) get an emoji marker instead of a bare
slug.
This commit is contained in:
2026-09-23 07:12:15 +00:00
parent 4f97a6592c
commit 97bde49296
4 changed files with 39 additions and 4 deletions
+7 -2
View File
@@ -9,7 +9,9 @@ from the kanta content database, path from `PAGERITE_ANALYTICS` (default:
`Favicon`), the `Store` (raw log + atomic JSON persistence) and
`Store.display()`, where **all** classification happens.
- `pagerite/pages.py` — records every served document as one raw GET line
(`_record_get`, in `pagerite/tracking.py`) with its true HTTP status.
(`_record_get`, in `pagerite/tracking.py`) with its true HTTP status, plus
the `/robots.txt` and `/sitemap.xml` machinery GETs (never followed by an
activity message, they surface as crawler hits).
- `pagerite/tracking.py` — the `/_ws` activity WebSocket, and
`WebSocket /_api/ws/analytics` (admin-gated like every `/_api` endpoint).
- `frontend/src/pagerite.js` — the client activity channel and the 📊 pen.
@@ -211,7 +213,10 @@ for misses.
are caught by the abuse rules regardless. In the viewer, crawler hits are
grouped by client hash and shown as a trail of pages, preceded by the
referer when there is one (rendered with its favicon like visit
referers). The crawler table lists the most recent crawler first, with
referers). Non-article machinery GETs (`/robots.txt`, `/sitemap.xml`
and feed paths such as `/rss.xml` once those routes exist) appear as
emoji-marked steps (🤖 / 🗺️ / 📡) so they stand out from article steps.
The crawler table lists the most recent crawler first, with
the most active as a tie-breaker.
- **Abuse (scanner) hits**: a 404 on a telltale path — an empty URL segment
(`//foo` — no real client generates those), any segment starting with a