Record robots.txt/sitemap.xml GETs, emoji-mark machinery steps in trails
The robots and sitemap routes never called _record_get, so those accesses were invisible. Recorded like page GETs, they surface as crawler hits at display time (no activity message ever follows). In the trail rendering, known non-article machinery paths (robots, sitemap, future feed paths) get an emoji marker instead of a bare slug.
This commit is contained in:
+7
-2
@@ -9,7 +9,9 @@ from the kanta content database, path from `PAGERITE_ANALYTICS` (default:
|
||||
`Favicon`), the `Store` (raw log + atomic JSON persistence) and
|
||||
`Store.display()`, where **all** classification happens.
|
||||
- `pagerite/pages.py` — records every served document as one raw GET line
|
||||
(`_record_get`, in `pagerite/tracking.py`) with its true HTTP status.
|
||||
(`_record_get`, in `pagerite/tracking.py`) with its true HTTP status, plus
|
||||
the `/robots.txt` and `/sitemap.xml` machinery GETs (never followed by an
|
||||
activity message, they surface as crawler hits).
|
||||
- `pagerite/tracking.py` — the `/_ws` activity WebSocket, and
|
||||
`WebSocket /_api/ws/analytics` (admin-gated like every `/_api` endpoint).
|
||||
- `frontend/src/pagerite.js` — the client activity channel and the 📊 pen.
|
||||
@@ -211,7 +213,10 @@ for misses.
|
||||
are caught by the abuse rules regardless. In the viewer, crawler hits are
|
||||
grouped by client hash and shown as a trail of pages, preceded by the
|
||||
referer when there is one (rendered with its favicon like visit
|
||||
referers). The crawler table lists the most recent crawler first, with
|
||||
referers). Non-article machinery GETs (`/robots.txt`, `/sitemap.xml` —
|
||||
and feed paths such as `/rss.xml` once those routes exist) appear as
|
||||
emoji-marked steps (🤖 / 🗺️ / 📡) so they stand out from article steps.
|
||||
The crawler table lists the most recent crawler first, with
|
||||
the most active as a tie-breaker.
|
||||
- **Abuse (scanner) hits**: a 404 on a telltale path — an empty URL segment
|
||||
(`//foo` — no real client generates those), any segment starting with a
|
||||
|
||||
Reference in New Issue
Block a user