LeoVasanko/pagerite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add llms.txt, JSON Feed and RSS feed exports

Browse Source 
New pagerite/feeds.py: /llms.txt (Markdown site map with excerpts for
LLM agents), /feed.json (JSON Feed 1.1) and /feed.xml (RSS 2.0 +
atom:link), all carrying every published article with full content and
absolutized URLs. Linked from every page <head> (feed alternates +
llms-txt) and from the sitemap, recorded in analytics like page GETs
and emoji-marked in the trails (🧠 llms.txt, 📡 feeds).
This commit is contained in:
LeoVasanko
2026-09-23 07:25:08 +00:00
parent 97bde49296
commit 9ff22016d1
8 changed files with 187 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/analytics.md
+3 -3
View File
@@ -213,9 +213,9 @@ for misses.
are caught by the abuse rules regardless. In the viewer, crawler hits are
grouped by client hash and shown as a trail of pages, preceded by the
referer when there is one (rendered with its favicon like visit
referers). Non-article machinery GETs (`/robots.txt`, `/sitemap.xml` —
and feed paths such as `/rss.xml` once those routes exist) appear as
emoji-marked steps (🤖 / 🗺️ / 📡) so they stand out from article steps.
referers). Non-article machinery GETs (`/robots.txt`, `/sitemap.xml`,
`/llms.txt`, the feeds) appear as
emoji-marked steps (🤖 / 🗺️ / 🧠 / 📡) so they stand out from article steps.
The crawler table lists the most recent crawler first, with
the most active as a tie-breaker.
- **Abuse (scanner) hits**: a 404 on a telltale path — an empty URL segment
docs/backend.md
+1
View File
@@ -10,6 +10,7 @@ Thin FastAPI assembly: lifespan (open the kanta database, load the file store, t
- `files.py` — the `FileStore` and image derivative helpers, and the file routes: `/_api/files`, `/_f/`, `/_themes/`, `/_fonts/`, the favicon settings endpoints.
- `api.py` — the editor REST API and WebSockets: `/_api/pages`, `/_api/structure`, `/_api/settings`, `/_api/toggle-task`, `/_api/translations`, `/_api/ws/editor`, and the translator channel `/_translate/{clientkey}`.
- `tracking.py` — visit analytics: GeoIP, client enrichment, favicon fetching, debounced broadcasts, the `/_ws` activity socket, the admin stream `/_api/ws/analytics`, and the `/_a` viewer page.
- `feeds.py` — machine-readable site exports: `/llms.txt` (Markdown site map for LLM agents), `/feed.json` (JSON Feed 1.1) and `/feed.xml` (RSS 2.0 + atom:link), all carrying every published article with full content (relative URLs absolutized), linked from every page's `<head>` and from the sitemap, and recorded in analytics like page GETs.
- `pages.py` — the public content pages: `/`, `/sitemap.xml`, `/robots.txt` and the `/{path:path}` catch-all.
Route ordering is load-bearing and lives in `app.py`: the api/tracking/files routers are included BEFORE `frontend.route(app, "/")` is called — fastapi-vue inserts its file routes at the position where `route()` was called (during `load()` in the lifespan), so anything registered earlier wins. The content catch-all `/{path:path}` is included AFTER `frontend.route()` so that built frontend assets still take priority over content slugs. The `Frontend` is constructed with `spa=False` explicitly: it only serves the built files without a catch-all.