Serve images extension-less at /_f/{hash} with Accept-negotiated AVIF/WebP/JPEG
Uploaded images (SVGs rasterized, GIFs excepted) are stored as the
original (hash.orig.ext, internal only, never served) plus AVIF primary
and WebP/JPEG fallback derivatives re-encoded from it. Pages link the
bare hash; the server serves a format only when Accept lists it
explicitly (image/avif -> AVIF, image/webp -> WebP, else JPEG) with
vary: accept, while an explicit extension pins the format. Favicons go
through the same pipeline at 192px. migrate_v2 rewrites old
/_f/{hash}.avif article links, a startup backfill creates missing
derivatives, and twitter:image pins the .webp variant for X's scraper.
This commit is contained in:
+2
-2
@@ -10,7 +10,7 @@ The build mirrors the URL space — hashed immutable assets under `/_assets/`, `
|
||||
|
||||
Generated HTML pages (content pages, category/404 placeholders, `/_a`) go through `_html_response`: zstd-compressed per request at level 9 when the client sends `accept-encoding: zstd` (no gzip fallback; static assets are pre-compressed by the `Frontend`), with `vary: accept-encoding` set and the ETag kept identical across encodings so `if-none-match` revalidation still works. In production the rendered bodies are cached in an LRU keyed by everything the output depends on — page kind, path, the site origin (social meta), encoding, and `data.version`, which bumps on every content/settings change and so transparently invalidates the whole cache. The cache is bypassed in dev, where theme/design CSS is re-read from disk per request. Content pages carry an ETag built from the node's modified timestamp and `data.version`; `/_a` instead gets a blake3 hash of the rendered body (it has no Node), with matching `if-none-match` revalidations answered by a 304.
|
||||
|
||||
Uploaded files, seed assets and fetched external-site favicons live in the `FileStore`: content-addressed files on disk under `<hostname>/files/` (`PAGERITE_FILES`), fully cached in RAM at startup — both the raw body and a zstd-compressed copy (kept only when smaller). `GET /_f/{name}` serves from the RAM cache with immutable caching, answering the zstd variant when the client accepts it; the name is the ETag. Legacy databases that still carry blobs in a `files` kanta field are migrated to disk by `pagerite/migrations.py::migrate_v1` (kanta's `migrate_vN` mechanism, wired via `Kanta(..., migrations="pagerite.migrations")`), which pops the field from the raw state before struct decoding.
|
||||
Uploaded files, seed assets and fetched external-site favicons live in the `FileStore`: content-addressed files on disk under `<hostname>/files/` (`PAGERITE_FILES`), fully cached in RAM at startup — both the raw body and a zstd-compressed copy (kept only when smaller). `GET /_f/{name}` serves from the RAM cache with immutable caching, answering the zstd variant when the client accepts it; the name is the ETag. Uploaded raster images (and rasterized SVGs) are stored as `<hash>.orig<ext>` (internal only, never served) plus AVIF, WebP and JPEG derivatives, and pages link the extension-less `/_f/{hash}`: the server serves a format only when the Accept header lists it explicitly (`image/avif` → AVIF, `image/webp` → WebP, otherwise — including `*/*` — JPEG), with `vary: accept`; an explicit extension pins the format. Missing derivatives of older uploads are backfilled at startup; `migrate_v2` rewrites old `/_f/{hash}.avif` article links to the bare form. Legacy databases that still carry blobs in a `files` kanta field are migrated to disk by `pagerite/migrations.py::migrate_v1` (kanta's `migrate_vN` mechanism, wired via `Kanta(..., migrations="pagerite.migrations")`), which pops the field from the raw state before struct decoding.
|
||||
|
||||
## `data.py`
|
||||
|
||||
@@ -26,7 +26,7 @@ markdown-it-py renderer (html passthrough + attrs, footnote, deflist, tasklists,
|
||||
|
||||
The shared page layout as an html5tagger `Template` with placeholders (`Title`, `Brand`, `Banner`, `Nav`, `Sidebar`, `Main`), nav rendering straight from the `Data.menu` tree (siblings sorted by `Node.order`; nav links to content-less labels point at their first child via `first_leaf`, the first published descendant with content), and page/404 rendering.
|
||||
|
||||
Content pages get SEO/social meta (description, canonical link, Open Graph + twitter card) from heuristics over the rendered article: the description is the first paragraph's text, the share image prefers a `{.hero}`-classed image, then the first raster `<img>`, then the first SVG; the first `<video>` yields `og:video`; URLs are made absolute with the site origin (`SITE_URL` — `https://<hostname>` from the CLI hostname argument; on localhost the request's own base URL is the fallback); `article:published/modified_time` come from `Node.created`/`modified`. The page title is injected as `# {title}` when the markdown has no h1 of its own, so it never appears twice (it always supplies `<title>` and nav labels).
|
||||
Content pages get SEO/social meta (description, canonical link, Open Graph + twitter card) from heuristics over the rendered article: the description is the first paragraph's text, the share image prefers a `{.hero}`-classed image, then the first raster `<img>`, then the first SVG; the first `<video>` yields `og:video`; URLs are made absolute with the site origin (`SITE_URL` — `https://<hostname>` from the CLI hostname argument; on localhost the request's own base URL is the fallback); `article:published/modified_time` come from `Node.created`/`modified`. Additionally `twitter:image` pins extension-less `/_f/{hash}` share images to the `.webp` variant — X only honors WebP via twitter:image (not og:image) and its scraper cannot be trusted to negotiate via Accept. The page title is injected as `# {title}` when the markdown has no h1 of its own, so it never appears twice (it always supplies `<title>` and nav labels).
|
||||
|
||||
The navbar holds top-level items only; the current section's subitems go to a left `#sidebar` as a nested list (the section's direct children plain, deeper levels indented with article-list-style markers), rendered only from the second level down — main-level pages list their children as cards after the content instead. Below that, the sidebar renders when the section offers at least two published items, or exactly one while viewing anything other than that only page — the section index, a 404, a grandchild (so those pages can reach the child), and also on that only page itself when it has published children of its own; no aside element at all on the front page, main-level pages, leaf pages and the sole childless page of a one-page section. Also, category labels are nodes without content — None *or* empty markdown — and their nav links point at their first child page. Dynamic regions have stable ids (`#page-banner`, `#nav`, `#sidebar`, `#main`) for fetch-navigation swaps (`#sidebar` may be absent on either side of a swap).
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ Siblings order by the fractional `Node.order` key: a moved item gets a fresh key
|
||||
|
||||
## Files
|
||||
|
||||
Files are content-addressed (blake3[:12] + extension) and stored **on disk** under `<hostname>/files/` (path from `PAGERITE_FILES`), served at `/_f/{name}` with immutable caching. Uploaded raster images (except SVG/GIF) are stored as a pair: the untouched original under `<hash>.orig<ext>` and a mediapreview-recompressed AVIF derivative (`<hash>.avif`, thumbnailed to `IMAGE_MAXSIZE` at `IMAGE_QUALITY`) which is the externally linked file; deleting either name removes the pair. the `FileStore` in app.py caches every file in RAM, both uncompressed and zstd-compressed (the compressed copy only when smaller), so `/_f` answers both encodings without disk reads. Pages reference files by absolute `/_f/` URLs so hierarchy moves never break them. Pre-refactor databases kept the blobs in a `Data.files` kanta field; the kanta migration `pagerite/migrations.py::migrate_v1` writes them to disk on open and drops the field (removed from `Data`). Fetched favicons of external analytics sites live in the same store (see `docs/analytics.md`).
|
||||
Files are content-addressed (blake3[:12] + extension) and stored **on disk** under `<hostname>/files/` (path from `PAGERITE_FILES`), served at `/_f/{name}` with immutable caching. Uploaded raster images (except GIF) and SVGs (rasterized) get a set of derivatives: the untouched original under `<hash>.orig<ext>` (internal only — it may carry EXIF data and is never served; SVG originals stay servable as `<hash>.svg`), a mediapreview-recompressed AVIF (`<hash>.avif`, thumbnailed to `IMAGE_MAXSIZE` at `IMAGE_QUALITY`), and WebP/JPEG fallbacks re-encoded from the AVIF at lower quality (`IMAGE_WEBP_QUALITY`/`IMAGE_JPG_QUALITY`, chosen for similar-or-smaller file size). Pages link the bare `/_f/<hash>` and the server negotiates by Accept header: a format is served only when listed explicitly (`image/avif` → AVIF, `image/webp` → WebP, anything else including `image/*` and `*/*` → JPEG); an explicit extension in the URL pins the format. Responses carry `vary: accept`. Favicons uploaded in settings go through the same pipeline at `FAVICON_MAXSIZE` (192px). Existing databases are updated by `migrate_v2` (link rewrite) plus a startup backfill that creates missing derivatives. Deleting any name of a hash removes the whole group. The `FileStore` in app.py caches every file in RAM, both uncompressed and zstd-compressed (the compressed copy only when smaller), so `/_f` answers both encodings without disk reads. Pages reference files by absolute `/_f/` URLs so hierarchy moves never break them. Pre-refactor databases kept the blobs in a `Data.files` kanta field; the kanta migration `pagerite/migrations.py::migrate_v1` writes them to disk on open and drops the field (removed from `Data`). Fetched favicons of external analytics sites live in the same store (see `docs/analytics.md`).
|
||||
|
||||
## Banners
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ Pagerite is a single-user CMS/blog. This document records the initial high-level
|
||||
- Content is written in **Markdown** with powerful extensions (tables, footnotes, code highlighting, etc.).
|
||||
- **Embedded HTML is passed through unfiltered**, including inline scripts and other dynamic content the author wants to post. This is safe by the single-trusted-author assumption above.
|
||||
- Renderer: **markdown-it-py** with mdit-py-plugins (footnotes, definition lists, task lists, brace-attributes, admonitions and `::: name` containers — generic `<div class="name">` wrappers (the name may be followed by brace attributes: `::: aside {.right}`), of which `::: aside` floats as a muted side box and `{.margin}` / `::: margin` marks any block a margin note — on all but phone widths they float in the side zone at the article's left (the region the nav sidebar overlays, or the sidebar's own track when the layout reserves one) and the text never moves — and `::: nocols` opts its section out of column layout; tables and strikethrough from the default preset), GitHub-style alerts (`> [!NOTE]` / TIP / IMPORTANT / WARNING / CAUTION, rendered in the admonition callout styling), with `html=True` for raw passthrough, `typographer=True` for SmartyPants-style replacements in body text (curly quotes, `--` / `---` → en / em dashes, `...` → ellipsis, `(c)` → ©, etc.), and `breaks=True` so single line breaks inside paragraphs become `<br>` — including inside blockquotes, where every newline is kept and a blank `>` line starts a new paragraph. Code spans/blocks and raw HTML are left untouched. Fenced code blocks are highlighted server-side with **Pygments** (`nowrap` spans styled by `/_assets/pygments-*.css`, which maps every token class onto the `--code-*` variables; the base stylesheet defines light and dark palette sets resolved via `light-dark()`, so each theme gets the set matching its `color-scheme` and may only retint `--code-bg` to keep the well in the page's color family); a JS copy button appears on hover. Should this prove limiting, we implement our own renderer on top of html5tagger, which we already use for all HTML generation.
|
||||
- **Files are content-addressed.** Uploads (`PUT /_api/files/{filename}`) are stored on disk (`<hostname>/files/`, RAM-cached uncompressed + zstd) by content hash — blake3, first 6 bytes hex + original extension — and served immutable from `/_f/{hash}.ext`. Raster images (not SVG/GIF) are recompressed via mediapreview: the original is kept as `/_f/{hash}.orig{ext}` while pages link the thumbnailed AVIF derivative `/_f/{hash}.avif`. Absolute URLs that survive page renames and dedupe identical content; pages no longer own files. An image standing alone in its paragraph becomes a block `<figure>` — with `<figcaption>` when it has a title; images inline with text and raw `<img>` HTML stay plain inline images. Positioning is by attribute classes: `{.right}` — `{.right}`, `{.left}` float at 30% of the text column (the caption wraps within it; an explicit `width=300` makes the figure shrink-wrap the image instead), `{.margin}` makes it a margin note, floating in the side zone left of the text on all but phone widths, `{.wide}` goes full bleed (viewport edge to edge, or up to the docked editor; the sidebar stacks on top of it); plain attributes like `width=300` work too. The same brace syntax on a block's last line (no blank line between) applies to the whole block: a paragraph ending with `{.wide}` becomes a full-width element that breaks out of the column layout, and space-separated at the end of a text line (`some text {.small}`) the braces likewise belong to the block — a space is what keeps them off an image or link ending the line, which keep their own directly-attached attrs; text size classes `{.small}` / `{.large}` / `{.huge}` (em-based) work on any block; written on the line after a block it applies to that preceding block — this is how headings, `::: containers` and code fences take classes (a wide code fence goes full bleed like a wide figure). Headings (h1/h2) clear floats, so images never overflow into the next section.
|
||||
- **Files are content-addressed.** Uploads (`PUT /_api/files/{filename}`) are stored on disk (`<hostname>/files/`, RAM-cached uncompressed + zstd) by content hash — blake3, first 6 bytes hex + original extension — and served immutable from `/_f/…`. Raster images (not GIF) and SVGs (rasterized) are recompressed via mediapreview: the original is kept as `{hash}.orig{ext}` (internal only, never served — it may carry EXIF data; SVG originals stay servable as `{hash}.svg`) while pages link the extension-less `/_f/{hash}` and the server picks from the derivatives (`{hash}.avif` / `{hash}.webp` / `{hash}.jpg`) by Accept header — a format only when listed explicitly (`image/avif` → AVIF, `image/webp` → WebP, otherwise JPEG), with `vary: accept`; an explicit extension in the URL pins the format. Absolute URLs that survive page renames and dedupe identical content; pages no longer own files. An image standing alone in its paragraph becomes a block `<figure>` — with `<figcaption>` when it has a title; images inline with text and raw `<img>` HTML stay plain inline images. Positioning is by attribute classes: `{.right}` — `{.right}`, `{.left}` float at 30% of the text column (the caption wraps within it; an explicit `width=300` makes the figure shrink-wrap the image instead), `{.margin}` makes it a margin note, floating in the side zone left of the text on all but phone widths, `{.wide}` goes full bleed (viewport edge to edge, or up to the docked editor; the sidebar stacks on top of it); plain attributes like `width=300` work too. The same brace syntax on a block's last line (no blank line between) applies to the whole block: a paragraph ending with `{.wide}` becomes a full-width element that breaks out of the column layout, and space-separated at the end of a text line (`some text {.small}`) the braces likewise belong to the block — a space is what keeps them off an image or link ending the line, which keep their own directly-attached attrs; text size classes `{.small}` / `{.large}` / `{.huge}` (em-based) work on any block; written on the line after a block it applies to that preceding block — this is how headings, `::: containers` and code fences take classes (a wide code fence goes full bleed like a wide figure). Headings (h1/h2) clear floats, so images never overflow into the next section.
|
||||
|
||||
## Page structure and navigation
|
||||
|
||||
@@ -48,7 +48,7 @@ Pagerite is a single-user CMS/blog. This document records the initial high-level
|
||||
- **Page mode** — the 🖊️ next to a page's heading (including 404s, which is how new pages start) opens a CodeMirror Markdown editor docked to the left of the article: the panel is fixed to the viewport's left edge (its top tracks the banner's bottom until the banner scrolls away), the content shifts right and the sidebar hides while editing. Preview renders server-side per keystroke (no debouncing) and swaps the whole visible article content in one go (the edit pen and category cards survive the swap).
|
||||
- **Site mode** — the ⚙️ at the top right (after the 📊 analytics link, before login) opens a panel with the site **brand** (applied to the header live), a **theme** selector (swapping the theme stylesheet in place), a **page transition** selector (`cube`/`crossfade`, swapping `#pagerite-transition` in place), **font** picks (heading/body/brand — stored as plain `:root` rows inside the custom CSS, referencing the base stylesheet's per-family font variables), a **site-wide custom CSS** field (injected into `<style id="pagerite-user">` in the live page head and swapped during fetch-navigation), the page's **banner design** selector (inherit / none / any design found on disk, inherited by children), the page's **banner HTML** field (supplementing the design, previewed into the real banner region, so you see exactly which banner you're editing) and the **structure tree**. Everything saves immediately as you edit — no save button, no edit mode.
|
||||
- Clicking a pen again closes the editor (without saving; a dirty preview reloads the page). The pens are `<button>`s wired up by `pagerite.js` — editing is an action, not a navigation. The editor's WebSocket **reconnects automatically** with local text and pending saves preserved. (All users are trusted authors for now; access control later with SSO.)
|
||||
- **CodeMirror 6** for Markdown editing (no WYSIWYG), title/published controls. Images can be pasted straight into the editor or chosen via a file input: they upload to the content store (`PUT /_api/files/...`) and insert `` at the cursor.
|
||||
- **CodeMirror 6** for Markdown editing (no WYSIWYG), title/published controls. Images can be pasted straight into the editor or chosen via a file input: they upload to the content store (`PUT /_api/files/...`) and insert `` at the cursor.
|
||||
- The **structure panel** (vue-draggable tree of the whole site, in site mode) covers page management: reorder any menu level, drag across sections, add, delete (two clicks: the button arms, then deletes — no dialogs). Every node is a real label — content-less category rows offer a ➕ to give them a landing page. Deleting a category removes only its landing page (the label and its subpages stay). Every non-empty list ends with a ➕ row that starts a new page as a local-only tree row at that level; the row can be dragged into place before its title and slug are filled in and is persisted only on commit. While dragging, these ➕ rows double as "end of this list" drop targets; dropping ON the lower part of a row makes the page that row's first child (even a leaf's, creating a sublist), while a row's exposed top edge inserts a sibling before it. A dragged row's indentation previews the target list's depth. Rows are always editable: titles save while typing, slug edits commit on blur/Enter since they rename the path (moving the whole subtree). The front page is the root row with an empty slug — renaming it away leaves no front page ("/" redirects to the first nav item), and giving another top-level row the empty slug makes it the front page.
|
||||
- Preview and saving go over a **WebSocket** (`/_api/ws/editor`) with a stateless JSON protocol (`open`/`render`/`save`; on save all fields are optional and absent ones keep their old values, `move_from` renames), avoiding REST polling and races. Rendering always stays server-side.
|
||||
- A REST API also exists for scripting, all under `/_api/`: `GET pages` (the full tree), `PUT/DELETE pages/{path}`, `GET/PUT settings` (site brand, theme and custom CSS), `POST structure` (reorder/move/retitle), file upload/removal via `PUT/DELETE files/{name}`.
|
||||
|
||||
+185
-31
@@ -80,10 +80,18 @@ analytics_store = analytics.Store(ANALYTICS_PATH)
|
||||
# files on disk under hash-prefixed names, cached in RAM, served at /_f/.
|
||||
FILES_DIR = Path(os.getenv("PAGERITE_FILES", str(SITE_DIR / "files")))
|
||||
|
||||
# Uploaded raster images are thumbnailed to this size and recompressed to
|
||||
# AVIF; the untouched original is kept alongside as ``<hash>.orig<ext>``.
|
||||
# Uploaded images are thumbnailed to this size and recompressed to AVIF
|
||||
# (primary), with WebP and JPEG fallbacks re-encoded from the AVIF at
|
||||
# somewhat lower quality (similar or smaller file size); the untouched
|
||||
# original is kept alongside as ``<hash>.orig<ext>`` (never served).
|
||||
IMAGE_MAXSIZE = 1920
|
||||
IMAGE_QUALITY = 60
|
||||
IMAGE_WEBP_QUALITY = 50
|
||||
IMAGE_JPG_QUALITY = 55
|
||||
|
||||
# Favicons get the same derivatives but thumbnailed much smaller — 192px
|
||||
# is plenty (browsers scale down for the 16x16 tab icon themselves).
|
||||
FAVICON_MAXSIZE = 192
|
||||
|
||||
# Live WebSocket clients for the analytics stream.
|
||||
_analytics_ws_clients: set[WebSocket] = set()
|
||||
@@ -199,9 +207,21 @@ def _hash_name(body: bytes, orig: str) -> str:
|
||||
|
||||
|
||||
def _store_seed_file(markdown: str, banner: str, orig: str, body: bytes) -> tuple[str, str]:
|
||||
"""Store a seed file content-addressed and point references at /_f/."""
|
||||
name = _hash_name(body, orig)
|
||||
file_store.put(name, body)
|
||||
"""Store a seed file content-addressed and point references at /_f/.
|
||||
|
||||
Images get the same AVIF/WebP/JPEG derivatives as uploads and are
|
||||
linked extension-less; other content is stored as-is with its
|
||||
extension."""
|
||||
digest = blake3.blake3(body).hexdigest()[:12]
|
||||
derivatives = None if _ext(orig) == ".gif" else _image_derivatives(body, _ext(orig))
|
||||
if derivatives is None:
|
||||
file_store.put(digest + _ext(orig), body)
|
||||
name = digest + _ext(orig)
|
||||
else:
|
||||
file_store.put(f"{digest}.svg" if _ext(orig) == ".svg" else f"{digest}.orig{_ext(orig)}", body)
|
||||
for fmt, variant in derivatives.items():
|
||||
file_store.put(f"{digest}.{fmt}", variant)
|
||||
name = digest
|
||||
markdown = markdown.replace(f"]({orig}", f"](/_f/{name}")
|
||||
banner = banner.replace(f'src="/{orig}"', f'src="/_f/{name}"')
|
||||
banner = banner.replace(f'src="{orig}"', f'src="/_f/{name}"')
|
||||
@@ -278,11 +298,51 @@ def _seed(data: Data) -> None:
|
||||
node.order = order
|
||||
|
||||
|
||||
def _backfill_derivatives() -> None:
|
||||
"""Create missing AVIF/WebP/JPEG derivatives for files stored before
|
||||
they were introduced (older uploads may have only the original plus
|
||||
AVIF, and SVGs no raster variants at all). WebP/JPEG are re-encoded
|
||||
from an existing AVIF when available, everything else from the
|
||||
original (SVGs rasterized first)."""
|
||||
try:
|
||||
paths = [f for f in file_store.path.iterdir() if f.is_file()]
|
||||
except FileNotFoundError:
|
||||
return
|
||||
groups: dict[str, list[Path]] = {}
|
||||
for p in paths:
|
||||
groups.setdefault(p.name.partition(".")[0], []).append(p)
|
||||
for digest, files in groups.items():
|
||||
names = {p.name for p in files}
|
||||
source = next(
|
||||
(p for p in files if ".orig." in p.name or p.suffix == ".svg"), None
|
||||
)
|
||||
if source is None:
|
||||
continue # plain as-is file, no derivatives to make
|
||||
avif = file_store.get(f"{digest}.avif")
|
||||
if avif is None:
|
||||
ext = source.suffix
|
||||
body = source.read_bytes()
|
||||
if ext == ".svg":
|
||||
png = _svg_to_png(body, IMAGE_MAXSIZE)
|
||||
if png is None:
|
||||
continue
|
||||
body, ext = png, ".png"
|
||||
converted = _to_avif(body, ext)
|
||||
if converted is None:
|
||||
continue
|
||||
file_store.put(f"{digest}.avif", converted)
|
||||
avif = file_store.get(f"{digest}.avif")
|
||||
for fmt, quality in (("webp", IMAGE_WEBP_QUALITY), ("jpg", IMAGE_JPG_QUALITY)):
|
||||
if f"{digest}.{fmt}" not in names:
|
||||
file_store.put(f"{digest}.{fmt}", _avif_to_format(avif[0], f".{fmt}", quality))
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
|
||||
"""Open the database, migrate legacy content, load assets, load GeoIP."""
|
||||
await kanta.open()
|
||||
await asyncio.to_thread(file_store.load)
|
||||
await asyncio.to_thread(_backfill_derivatives)
|
||||
_migrate_legacy()
|
||||
await frontend.load()
|
||||
# Decompress/open the DB-IP MMDB once at startup. Lookups are then
|
||||
@@ -361,11 +421,11 @@ class FileStore:
|
||||
self._cache[name] = self._entry(body)
|
||||
|
||||
def delete(self, name: str) -> None:
|
||||
"""Delete a file plus its derivative/original counterpart, if any.
|
||||
"""Delete a file plus its derivatives/original counterparts, if any.
|
||||
|
||||
An image upload is stored as a pair sharing the hash prefix
|
||||
(``<hash>.orig.<ext>`` + ``<hash>.avif``); deleting either removes
|
||||
both.
|
||||
An image upload is stored as a group sharing the hash prefix
|
||||
(``<hash>.orig.<ext>`` + ``<hash>.avif/.webp/.jpg``); deleting any
|
||||
of the names removes them all.
|
||||
"""
|
||||
stem = name.partition(".")[0]
|
||||
for key in [k for k in self._cache if k.partition(".")[0] == stem]:
|
||||
@@ -611,15 +671,27 @@ async def put_settings(settings: SettingsIn) -> None:
|
||||
async def put_favicon(request: Request) -> dict[str, str]:
|
||||
"""Upload a favicon into the content-addressed store and activate it.
|
||||
|
||||
Raw image body (ico/png/svg...); the stored name is a blake3 hash
|
||||
prefix + extension, and pages link it as <link rel="icon">. Returns
|
||||
Raw image body (ico/png/svg...). Decodable images are thumbnailed to
|
||||
FAVICON_MAXSIZE (192px — browsers scale down from there themselves)
|
||||
and stored as AVIF/WebP/JPEG derivatives linked extension-less; SVG
|
||||
originals also stay servable under their ``.svg`` name. Undecodable
|
||||
bodies are stored as-is. Pages link it as <link rel="icon">. Returns
|
||||
{"path": "/_f/..."}.
|
||||
"""
|
||||
body = await request.body()
|
||||
if not body:
|
||||
raise HTTPException(400, "empty file")
|
||||
stored = _hash_name(body, request.headers.get("x-filename", "favicon.ico"))
|
||||
ext = _ext(request.headers.get("x-filename", "favicon.ico"))
|
||||
digest = blake3.blake3(body).hexdigest()[:12]
|
||||
derivatives = await asyncio.to_thread(_image_derivatives, body, ext, FAVICON_MAXSIZE)
|
||||
if derivatives is None: # undecodable (e.g. some .ico): store as-is
|
||||
stored = digest + ext
|
||||
file_store.put(stored, body)
|
||||
else:
|
||||
stored = digest
|
||||
file_store.put(f"{digest}.svg" if ext == ".svg" else f"{digest}.orig{ext}", body)
|
||||
for fmt, variant in derivatives.items():
|
||||
file_store.put(f"{digest}.{fmt}", variant)
|
||||
with kanta.transaction("upload favicon"):
|
||||
data.favicon = stored
|
||||
data.version += 1
|
||||
@@ -675,7 +747,7 @@ async def toggle_task_endpoint(body: ToggleTaskIn) -> dict[str, str]:
|
||||
return {"markdown": new_markdown}
|
||||
|
||||
|
||||
def _to_avif(body: bytes, ext: str) -> bytes | None:
|
||||
def _to_avif(body: bytes, ext: str, maxsize: int = IMAGE_MAXSIZE) -> bytes | None:
|
||||
"""Recompress an image body to a thumbnailed AVIF via mediapreview's
|
||||
dispatch (pyvips for common formats, ffmpeg for HEIC/HEIF/AVIF), or
|
||||
None if the body is not a decodable image (stored as-is by the caller).
|
||||
@@ -689,7 +761,7 @@ def _to_avif(body: bytes, ext: str) -> bytes | None:
|
||||
avif, _resp = dispatch(
|
||||
Path(tmp.name),
|
||||
quality=IMAGE_QUALITY,
|
||||
maxsize=IMAGE_MAXSIZE,
|
||||
maxsize=maxsize,
|
||||
maxzoom=1,
|
||||
)
|
||||
except Exception:
|
||||
@@ -697,6 +769,56 @@ def _to_avif(body: bytes, ext: str) -> bytes | None:
|
||||
return avif
|
||||
|
||||
|
||||
def _svg_to_png(body: bytes, maxsize: int) -> bytes | None:
|
||||
"""Rasterize an SVG to PNG via pyvips, scaled so the long side is
|
||||
``maxsize`` — SVGs often carry no meaningful intrinsic resolution, so
|
||||
we rasterize at full image size rather than the tiny nominal one."""
|
||||
import pyvips
|
||||
|
||||
try:
|
||||
img = pyvips.Image.new_from_buffer(body, "")
|
||||
scale = maxsize / max(img.width, img.height) if img.width and img.height else maxsize
|
||||
if scale != 1:
|
||||
img = pyvips.Image.new_from_buffer(body, "", scale=scale)
|
||||
return img.write_to_buffer(".png")
|
||||
except pyvips.Error:
|
||||
return None
|
||||
|
||||
|
||||
def _avif_to_format(avif: bytes, suffix: str, quality: int) -> bytes:
|
||||
"""Re-encode the AVIF derivative into a fallback format (WebP/JPEG)
|
||||
via pyvips. JPEG has no alpha, so it is flattened onto white;
|
||||
``strip`` keeps metadata (EXIF) out of the fallbacks."""
|
||||
import pyvips
|
||||
|
||||
img = pyvips.Image.new_from_buffer(avif, "")
|
||||
if suffix == ".jpg" and img.hasalpha():
|
||||
img = img.flatten(background=[255, 255, 255])
|
||||
return img.write_to_buffer(suffix, Q=quality, strip=True)
|
||||
|
||||
|
||||
def _image_derivatives(body: bytes, ext: str, maxsize: int = IMAGE_MAXSIZE) -> dict[str, bytes] | None:
|
||||
"""The served variants of an uploaded image: ``avif`` (primary,
|
||||
thumbnailed to ``maxsize``) plus ``webp`` and ``jpg`` fallbacks
|
||||
re-encoded from it. SVGs are rasterized first (they are vector, so
|
||||
the raster replaces nothing — the .svg itself stays servable).
|
||||
Returns None for non-decodable content (stored as-is by the caller).
|
||||
"""
|
||||
if ext == ".svg":
|
||||
png = _svg_to_png(body, maxsize)
|
||||
if png is None:
|
||||
return None
|
||||
body, ext = png, ".png"
|
||||
avif = _to_avif(body, ext, maxsize)
|
||||
if avif is None:
|
||||
return None
|
||||
return {
|
||||
"avif": avif,
|
||||
"webp": _avif_to_format(avif, ".webp", IMAGE_WEBP_QUALITY),
|
||||
"jpg": _avif_to_format(avif, ".jpg", IMAGE_JPG_QUALITY),
|
||||
}
|
||||
|
||||
|
||||
@app.put("/_api/files/{name}")
|
||||
async def upload_file(name: str, request: Request) -> dict[str, str]:
|
||||
"""Store an upload (image, video...) in the content-addressed store.
|
||||
@@ -704,11 +826,14 @@ async def upload_file(name: str, request: Request) -> dict[str, str]:
|
||||
The stored name is a blake3 hash prefix + the original extension,
|
||||
served immutable at "/_f/{name}"; returns {"path": "/_f/..."}.
|
||||
|
||||
Raster images are additionally recompressed with mediapreview: the
|
||||
original goes to ``<hash>.orig<ext>`` (kept for reprocessing) while
|
||||
pages link the thumbnailed AVIF derivative ``<hash>.avif``. SVGs and
|
||||
GIFs are stored as-is (vector/animation would be lost). Other content
|
||||
and failed conversions fall back to plain storage.
|
||||
Raster images and SVGs are recompressed (SVGs rasterized) into AVIF
|
||||
(primary) plus WebP and JPEG fallbacks: the original goes to
|
||||
``<hash>.orig<ext>`` (kept for reprocessing, never served — it may
|
||||
carry EXIF data; SVG originals stay servable as ``<hash>.svg`` since
|
||||
vector carries no EXIF) and pages link the bare ``/_f/<hash>``, the
|
||||
server picking the format from the request's Accept header. GIFs are
|
||||
stored as-is (animation would be lost), as is other non-decodable
|
||||
content.
|
||||
"""
|
||||
if "/" in name or name in {".", ".."}:
|
||||
raise HTTPException(400, "bad file name")
|
||||
@@ -717,18 +842,19 @@ async def upload_file(name: str, request: Request) -> dict[str, str]:
|
||||
raise HTTPException(400, "empty file")
|
||||
ext = _ext(name)
|
||||
digest = blake3.blake3(body).hexdigest()[:12]
|
||||
avif = (
|
||||
derivatives = (
|
||||
None
|
||||
if ext in {".svg", ".gif"}
|
||||
else await asyncio.to_thread(_to_avif, body, ext)
|
||||
if ext == ".gif"
|
||||
else await asyncio.to_thread(_image_derivatives, body, ext)
|
||||
)
|
||||
if avif is None: # not a decodable image: store the body as-is
|
||||
if derivatives is None: # not a decodable image: store the body as-is
|
||||
stored = digest + ext
|
||||
file_store.put(stored, body)
|
||||
return {"path": f"/_f/{stored}"}
|
||||
file_store.put(f"{digest}.orig{ext}", body)
|
||||
file_store.put(f"{digest}.avif", avif)
|
||||
return {"path": f"/_f/{digest}.avif"}
|
||||
file_store.put(f"{digest}.svg" if ext == ".svg" else f"{digest}.orig{ext}", body)
|
||||
for fmt, variant in derivatives.items():
|
||||
file_store.put(f"{digest}.{fmt}", variant)
|
||||
return {"path": f"/_f/{digest}"}
|
||||
|
||||
|
||||
@app.delete("/_api/files/{name}", status_code=204)
|
||||
@@ -786,19 +912,47 @@ async def stored_file(name: str, request: Request) -> Response:
|
||||
"""Serve a file from the content-addressed store (immutable: the name
|
||||
is its own hash, so cache forever). Bodies are served from the RAM
|
||||
cache, zstd-compressed when the client accepts it and compression
|
||||
actually shrank the file."""
|
||||
actually shrank the file.
|
||||
|
||||
A bare ``/_f/{hash}`` (no extension, how pages link uploaded images)
|
||||
content-negotiates between the stored derivatives: a format is served
|
||||
only when the Accept header lists it explicitly — ``image/avif`` →
|
||||
AVIF, ``image/webp`` → WebP, anything else (including ``image/*`` and
|
||||
``*/*``) → JPEG. An explicit extension pins the format. ``.orig.``
|
||||
originals are internal (they may carry EXIF data) and never served."""
|
||||
if ".orig." in name:
|
||||
raise HTTPException(404)
|
||||
etag = name
|
||||
vary = ""
|
||||
entry = file_store.get(name)
|
||||
if entry is None and "." not in name:
|
||||
# Extension-less image link: negotiate avif/webp/jpg by Accept.
|
||||
vary = "accept"
|
||||
accept = request.headers.get("accept", "")
|
||||
if "image/avif" in accept:
|
||||
order = ("avif", "webp", "jpg")
|
||||
elif "image/webp" in accept:
|
||||
order = ("webp", "jpg", "avif")
|
||||
else:
|
||||
order = ("jpg", "webp", "avif")
|
||||
for ext in order:
|
||||
etag = f"{name}.{ext}"
|
||||
entry = file_store.get(etag)
|
||||
if entry is not None:
|
||||
break
|
||||
if entry is None:
|
||||
raise HTTPException(404)
|
||||
if request.headers.get("if-none-match") == name:
|
||||
if request.headers.get("if-none-match") == etag:
|
||||
return Response(status_code=304)
|
||||
body, compressed = entry
|
||||
headers = {"etag": name, "cache-control": "public, max-age=31536000, immutable"}
|
||||
headers = {"etag": etag, "cache-control": "public, max-age=31536000, immutable"}
|
||||
if compressed is not None and "zstd" in request.headers.get("accept-encoding", ""):
|
||||
headers["content-encoding"] = "zstd"
|
||||
headers["vary"] = "accept-encoding"
|
||||
vary = f"{vary}, accept-encoding".lstrip(", ")
|
||||
body = compressed
|
||||
mime = mimetypes.guess_type(name)[0] or "application/octet-stream"
|
||||
if vary:
|
||||
headers["vary"] = vary
|
||||
mime = mimetypes.guess_type(etag)[0] or "application/octet-stream"
|
||||
return Response(body, media_type=mime, headers=headers)
|
||||
|
||||
|
||||
|
||||
@@ -6,6 +6,15 @@ per database based on its recorded version.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import re
|
||||
|
||||
#: Extension-less file links: uploaded images are now linked as /_f/<hash>
|
||||
#: and the server negotiates avif/webp from the Accept header.
|
||||
_DERIVATIVE_LINK = re.compile(r"(/_f/[0-9a-f]{12})\.(?:avif|webp)\b")
|
||||
|
||||
|
||||
def _rewrite_links(text: str | None) -> str | None:
|
||||
return None if text is None else _DERIVATIVE_LINK.sub(r"\1", text)
|
||||
|
||||
|
||||
def migrate_v1(d: dict) -> None:
|
||||
@@ -21,3 +30,21 @@ def migrate_v1(d: dict) -> None:
|
||||
if isinstance(body, str): # JSON-level bytes are base64 strings
|
||||
body = base64.b64decode(body)
|
||||
file_store.put(name, body)
|
||||
|
||||
|
||||
def _rewrite_links(text: str) -> str:
|
||||
return _DERIVATIVE_LINK.sub(r"\1", text)
|
||||
|
||||
|
||||
def migrate_v2(d: dict) -> None:
|
||||
"""Strip .avif/.webp extensions from /_f/ links in page content and
|
||||
banners (extension-less URLs negotiate the format by Accept header)."""
|
||||
|
||||
def walk(nodes: dict) -> None:
|
||||
for node in nodes.values():
|
||||
for field in ("content", "banner"):
|
||||
if isinstance(node.get(field), str):
|
||||
node[field] = _rewrite_links(node[field])
|
||||
walk(node.get("children") or {})
|
||||
|
||||
walk(d.get("menu") or {})
|
||||
|
||||
+1
-1
@@ -51,7 +51,7 @@ The 🖊️ pens open a tabbed editor over the page you are viewing:
|
||||
|
||||
The URL is the structure: a page at `docs/markdown` lives under `docs`, and the menus are derived from that. Slugs are lowercase ASCII (`a-z 0-9 - _`). A node without content is a category label — it renders a placeholder and its menu link points at its first child page. This site's own `docs` label demonstrates that, and the sidebar on this page shows the two submenu levels below it.
|
||||
|
||||
Images and files uploaded anywhere land in a content-addressed store served from `/_f/{hash}.ext`, so links survive page moves. The article editor's format bar and copy-paste both upload images for you.
|
||||
Images and files uploaded anywhere land in a content-addressed store served from `/_f/{hash}`, so links survive page moves. The server picks AVIF, WebP or JPEG from your browser's Accept header. The article editor's format bar and copy-paste both upload images for you.
|
||||
|
||||
{dates}
|
||||
"""
|
||||
|
||||
+12
-1
@@ -815,7 +815,10 @@ def _share_media(html: str, base_url: str) -> tuple[str, str]:
|
||||
"""(image, video) share URLs from the rendered article.
|
||||
|
||||
The _media picks as absolute URLs built from the request base —
|
||||
social scrapers cannot use relative ones.
|
||||
social scrapers cannot use relative ones. Extension-less store links
|
||||
(``/_f/<hash>``) are used as-is: the server negotiates the format
|
||||
from the scraper's Accept header (no explicit image/avif|webp → JPEG,
|
||||
which every scraper supports).
|
||||
"""
|
||||
if not base_url:
|
||||
return "", ""
|
||||
@@ -838,10 +841,17 @@ def _social_meta(
|
||||
share image the article's first <img> — authors lead with their most
|
||||
representative figure. Absolute URLs are built from the request's base
|
||||
(social scrapers cannot use relative ones).
|
||||
|
||||
``twitter:image`` pins extension-less store links to the ``.webp``
|
||||
variant: X only honors WebP via twitter:image (not og:image) and its
|
||||
scraper cannot be trusted to negotiate via Accept.
|
||||
"""
|
||||
url = f"{base_url}/{path}" if base_url else ""
|
||||
text = _description(html)
|
||||
image, video = _share_media(html, base_url)
|
||||
twitter_image = (
|
||||
re.sub(r"(/_f/[0-9a-f]{12})$", r"\1.webp", image) if image else ""
|
||||
)
|
||||
return {
|
||||
"description": text,
|
||||
"canonical": url,
|
||||
@@ -855,6 +865,7 @@ def _social_meta(
|
||||
"article:published_time": node.created.isoformat(),
|
||||
"article:modified_time": node.modified.isoformat(),
|
||||
"twitter:card": "summary_large_image" if image else "summary",
|
||||
"twitter:image": twitter_image,
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user