Add llms.txt, JSON Feed and RSS feed exports

New pagerite/feeds.py: /llms.txt (Markdown site map with excerpts for LLM agents), /feed.json (JSON Feed 1.1) and /feed.xml (RSS 2.0 + atom:link), all carrying every published article with full content and absolutized URLs. Linked from every page <head> (feed alternates + llms-txt) and from the sitemap, recorded in analytics like page GETs and emoji-marked in the trails (🧠 llms.txt, 📡 feeds).
2026-09-23 07:25:08 +00:00
parent 97bde49296
commit 9ff22016d1
8 changed files with 187 additions and 6 deletions
@@ -16,6 +16,7 @@ Pagerite is a CMS. See `docs` for the full design and implementation details. Ke
  - `api.py` — editor REST + WS: `/_api/pages`, `/_api/structure`, `/_api/settings`, `/_api/toggle-task`, `/_api/translations`, `/_api/ws/editor`, `/_translate/{key}`.
  - `tracking.py` — visit analytics: GeoIP, client enrichment, favicon fetch, `/_ws`, `/_api/ws/analytics`, the `/_a` page (docs/analytics.md).
  - `pages.py` — public content pages: `/`, `/sitemap.xml`, `/robots.txt`, the `/{path:path}` catch-all.
+  - `feeds.py` — machine-readable exports: `/llms.txt`, `/feed.json` (JSON Feed 1.1), `/feed.xml` (RSS 2.0 + atom:link); all published articles, full content, linked from every page `<head>` and the sitemap, recorded in analytics.
  - `data.py` — msgspec Structs for the kanta database.
  - `chunks.py` — block-level Markdown chunking and content-hash keys for the chunk stores (docs/migrate.md).
  - `i18n.py` — language selection, translation assembly (chunks + overrides) and translated-edit recording (per-chunk user overrides in `Data.overrides`, per-language title overrides, refresh).
@@ -213,9 +213,9 @@ for misses.
  are caught by the abuse rules regardless. In the viewer, crawler hits are
  grouped by client hash and shown as a trail of pages, preceded by the
  referer when there is one (rendered with its favicon like visit
-  referers). Non-article machinery GETs (`/robots.txt`, `/sitemap.xml` —
-  and feed paths such as `/rss.xml` once those routes exist) appear as
-  emoji-marked steps (🤖 / 🗺️ / 📡) so they stand out from article steps.
+  referers). Non-article machinery GETs (`/robots.txt`, `/sitemap.xml`,
+  `/llms.txt`, the feeds) appear as
+  emoji-marked steps (🤖 / 🗺️ / 🧠 / 📡) so they stand out from article steps.
  The crawler table lists the most recent crawler first, with
  the most active as a tie-breaker.
 - **Abuse (scanner) hits**: a 404 on a telltale path — an empty URL segment
@@ -10,6 +10,7 @@ Thin FastAPI assembly: lifespan (open the kanta database, load the file store, t
 - `files.py` — the `FileStore` and image derivative helpers, and the file routes: `/_api/files`, `/_f/`, `/_themes/`, `/_fonts/`, the favicon settings endpoints.
 - `api.py` — the editor REST API and WebSockets: `/_api/pages`, `/_api/structure`, `/_api/settings`, `/_api/toggle-task`, `/_api/translations`, `/_api/ws/editor`, and the translator channel `/_translate/{clientkey}`.
 - `tracking.py` — visit analytics: GeoIP, client enrichment, favicon fetching, debounced broadcasts, the `/_ws` activity socket, the admin stream `/_api/ws/analytics`, and the `/_a` viewer page.
+- `feeds.py` — machine-readable site exports: `/llms.txt` (Markdown site map for LLM agents), `/feed.json` (JSON Feed 1.1) and `/feed.xml` (RSS 2.0 + atom:link), all carrying every published article with full content (relative URLs absolutized), linked from every page's `<head>` and from the sitemap, and recorded in analytics like page GETs.
 - `pages.py` — the public content pages: `/`, `/sitemap.xml`, `/robots.txt` and the `/{path:path}` catch-all.

 Route ordering is load-bearing and lives in `app.py`: the api/tracking/files routers are included BEFORE `frontend.route(app, "/")` is called — fastapi-vue inserts its file routes at the position where `route()` was called (during `load()` in the lifespan), so anything registered earlier wins. The content catch-all `/{path:path}` is included AFTER `frontend.route()` so that built frontend assets still take priority over content slugs. The `Frontend` is constructed with `spa=False` explicitly: it only serves the built files without a catch-all.
@@ -167,10 +167,12 @@ function externalOrigin(url) {
 const MACHINE_STEPS = {
  '/robots.txt': ['🤖', 'robots.txt'],
  '/sitemap.xml': ['🗺️', 'sitemap.xml'],
+  '/llms.txt': ['🧠', 'llms.txt'],
+  '/feed.json': ['📡', 'feed.json'],
+  '/feed.xml': ['📡', 'feed.xml'],
  '/rss.xml': ['📡', 'rss.xml'],
  '/atom.xml': ['📡', 'atom.xml'],
  '/feed': ['📡', 'feed'],
-  '/feed.xml': ['📡', 'feed.xml'],
 }

 /** Format one trail step: an internal page or an external https origin. */
@@ -11,6 +11,8 @@ The routes live in specialized modules, included below as APIRouters:
  (``/_api/*``, ``/_translate/{clientkey}``).
 - ``pagerite.tracking`` — visit analytics (``/_ws``, ``/_api/ws/analytics``,
  the ``/_a`` viewer page).
+- ``pagerite.feeds`` — machine-readable site exports: ``/llms.txt``,
+  ``/feed.json`` (JSON Feed) and ``/feed.xml`` (RSS).
 - ``pagerite.pages`` — the public content pages: ``/``, ``/sitemap.xml``,
  ``/robots.txt`` and the ``/{path:path}`` catch-all.

@@ -39,7 +41,7 @@ from fastapi.responses import Response
 from fastapi_vue import Frontend, env
 from starlette.types import ASGIApp, Receive, Scope, Send

-from pagerite import api, files, pages, tracking
+from pagerite import api, feeds, files, pages, tracking
 from pagerite.files import file_store
 from pagerite.state import analytics_store, config, kanta

@@ -128,4 +130,5 @@ frontend.route(app, "/")

 # The content catch-all goes last: built assets win over content slugs,
 # anything unmatched falls through to content (and 404).
+app.include_router(feeds.router)
 app.include_router(pages.router)
@@ -0,0 +1,151 @@
+"""Machine-readable site exports: /llms.txt, /feed.json and /feed.xml.
+
+- ``/llms.txt`` (llmstxt.org convention): a Markdown map of the site for
+  LLM agents — the brand as title, then every published article as a link
+  with a short excerpt.
+- ``/feed.json``: JSON Feed 1.1 of all published articles, full content.
+- ``/feed.xml``: the same as RSS 2.0 (with an atom:link self reference)
+  for older feed readers.
+
+All three are linked from every page's <head> (see views._layout) and from
+the sitemap, recorded in analytics like page GETs (they surface as crawler
+hits — no activity message ever follows them), and rendered in the site's
+primary language only (feeds have no per-language negotiation here).
+"""
+
+import json
+from datetime import UTC
+from email.utils import format_datetime
+from xml.sax.saxutils import escape as xml_escape
+
+from fastapi import APIRouter, Request
+from fastapi.responses import Response
+
+from pagerite.data import Node, node_markdown, sorted_nodes
+from pagerite.markdown import make_md
+from pagerite.state import SITE_URL, data
+from pagerite.tracking import _record_get
+from pagerite.views import _description
+
+router = APIRouter()
+
+_md = make_md()
+
+
+def _articles() -> list[tuple[str, Node]]:
+    """All published content pages in menu order: (path, node)."""
+    out: list[tuple[str, Node]] = []
+
+    def walk(nodes: dict[str, Node], prefix: str) -> None:
+        for slug, node in sorted_nodes(nodes):
+            path = f"{prefix}/{slug}" if prefix else slug
+            if node.published and node.chunks is not None:
+                out.append((path, node))
+            if node.children:
+                walk(node.children, path)
+
+    walk(data.menu, "")
+    return out
+
+
+def _body_html(node: Node, base: str) -> str:
+    """Full article HTML for feed content, with relative URLs absolutized.
+
+    Rendered without the layout segmentation of page rendering (colseg
+    wrappers are meaningless in a feed reader); the item title carries the
+    page title, so no implicit h1 is injected either.
+    """
+    html = _md.render(node_markdown(data, node) or "")
+    return html.replace('src="/', f'src="{base}/').replace('href="/', f'href="{base}/')
+
+
+def _rfc822(node: Node) -> str:
+    return format_datetime(node.modified.astimezone(UTC), usegmt=True)
+
+
+def _iso(node: Node) -> str:
+    return node.modified.astimezone(UTC).replace(microsecond=0).isoformat()
+
+
+@router.get("/llms.txt")
+async def llms_txt(request: Request) -> Response:
+    """Markdown map of the site for LLM agents (llmstxt.org)."""
+    base = SITE_URL or str(request.base_url).rstrip("/")
+    lines = [f"# {data.brand}", "", "## Pages", ""]
+    for path, node in _articles():
+        url = f"{base}/{path}" if path else base
+        excerpt = _description(_body_html(node, base), 120)
+        suffix = f": {excerpt}" if excerpt else ""
+        lines.append(f"- [{node.title or path}]({url}){suffix}")
+    _record_get(request)
+    return Response(
+        "\n".join(lines) + "\n",
+        media_type="text/plain",
+        headers={"cache-control": "no-cache"},
+    )
+
+
+@router.get("/feed.json")
+async def feed_json(request: Request) -> Response:
+    """JSON Feed 1.1 of all published articles, full content."""
+    base = SITE_URL or str(request.base_url).rstrip("/")
+    items = [
+        {
+            "id": (url := f"{base}/{path}" if path else base),
+            "url": url,
+            "title": node.title or path,
+            "content_html": _body_html(node, base),
+            "date_published": node.created.astimezone(UTC)
+            .replace(microsecond=0)
+            .isoformat(),
+            "date_modified": _iso(node),
+        }
+        for path, node in _articles()
+    ]
+    feed = {
+        "version": "https://jsonfeed.org/version/1.1",
+        "title": data.brand,
+        "home_page_url": base,
+        "feed_url": f"{base}/feed.json",
+        "items": items,
+    }
+    _record_get(request)
+    return Response(
+        json.dumps(feed, ensure_ascii=False, indent=1),
+        media_type="application/feed+json",
+        headers={"cache-control": "no-cache"},
+    )
+
+
+@router.get("/feed.xml")
+async def feed_xml(request: Request) -> Response:
+    """RSS 2.0 of all published articles (full content in CDATA), with an
+    atom:link self reference."""
+    base = SITE_URL or str(request.base_url).rstrip("/")
+    items = []
+    for path, node in _articles():
+        url = f"{base}/{path}" if path else base
+        items.append(
+            f"<item><title>{xml_escape(node.title or path)}</title>"
+            f"<link>{xml_escape(url)}</link>"
+            f'<guid isPermaLink="true">{xml_escape(url)}</guid>'
+            f"<pubDate>{_rfc822(node)}</pubDate>"
+            f"<description><![CDATA[{_body_html(node, base)}]]></description>"
+            f"</item>"
+        )
+    xml = (
+        '<?xml version="1.0" encoding="UTF-8"?>'
+        '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">'
+        f"<channel><title>{xml_escape(data.brand)}</title>"
+        f"<link>{xml_escape(base)}</link>"
+        f"<description>{xml_escape(data.brand)}</description>"
+        f'<atom:link href="{xml_escape(base)}/feed.xml" rel="self" type="application/rss+xml" />'
+        + "".join(items)
+        + "</channel></rss>"
+    )
+    _record_get(request)
+    return Response(
+        xml,
+        media_type="application/rss+xml",
+        headers={"cache-control": "no-cache"},
+    )
@@ -55,7 +55,8 @@ async def front_page(request: Request) -> Response:

@router.get("/sitemap.xml")
 async def sitemap(request: Request) -> Response:
-    """Dynamically generate a sitemap of all published article pages."""
+    """Dynamically generate a sitemap of all published article pages, plus
+    the machine-readable exports (feeds, llms.txt)."""
    base = SITE_URL or str(request.base_url).rstrip("/")
    entries: list[tuple[str, datetime, int]] = []

@@ -112,6 +113,23 @@ async def sitemap(request: Request) -> Response:
        )
    lines.append("</urlset>")

+    # The machine-readable exports (feeds, llms.txt) are linked too, with
+    # the latest article's modification time as their lastmod.
+    if entries:
+        latest = max(m for _, m, _ in entries)
+        lastmod = (
+            latest.astimezone(UTC)
+            .replace(microsecond=0)
+            .isoformat()
+            .replace("+00:00", "Z")
+        )
+        for special in ("llms.txt", "feed.json", "feed.xml"):
+            lines.insert(
+                -1,
+                f"  <url><loc>{xml_escape(f'{base}/{special}')}</loc>"
+                f"<lastmod>{lastmod}</lastmod></url>",
+            )
+
    # Recorded like a page GET: never followed by an activity message, so
    # it lands in the crawler list at display time (docs/analytics.md).
    _record_get(request)
@@ -379,6 +379,11 @@ def _layout(
        doc.link(rel="canonical", href=canonical)
    for hreflang, href in alternates:
        doc.link(rel="alternate", hreflang=hreflang, href=href)
+    # Feed/LLM discovery links (see pagerite/feeds.py): identical on every
+    # page, so the positional <head> sync (swapdoc.js) is unaffected.
+    doc.link(rel="alternate", type="application/feed+json", title="JSON Feed", href="/feed.json")
+    doc.link(rel="alternate", type="application/rss+xml", title="RSS feed", href="/feed.xml")
+    doc.link(rel="llms-txt", type="text/markdown", title="llms.txt", href="/llms.txt")
    for key, value in (social or {}).items():
        if value:
            if key.startswith(("og:", "article:")):