paskia/docs/RemoteProxy.md

Remote Proxy / Satellite Design Review

Design notes for serving configured domains (rp-ids) from a remote paskia instance instead of the local database, so that latency-sensitive checks (/auth/api/forward, /auth/api/validate) answer in ~1 ms even when the auth server is on another continent. Example: app2.example.com runs on our local host and needs fast local checks, while app1.example.com and auth.example.com run far away — all sharing example.com as rp-id.

Client applications that previously used https://auth.example.com as their auth backend (forward-auth checks) only repoint to the local satellite (http://127.0.0.1:4401); both remain usable interchangeably, and the satellite ultimately uses auth.example.com.

Status: proposal, not implemented. Revision 2: no db.data() magic; explicit domain-aware stores; auth host mandatory for remote domains.

Facts from the architecture review

• The whole DB is already in RAM (msgspec structs, Kanta JSONL for persistence). Reads never touch disk — the remote is fast, so the latency problem is purely network RTT. The job is replicating read state, not reimplementing a database.
• /auth/api/forward (the hot path) is a pure read: session_ctx(cookie, host) → perm/max_age checks → 204 + Remote-* headers. No DB write, no cookie mutation.
• Sessions are host-bound and expire by validated + SESSION_LIFETIME; expiry is enforced by background deletion (db/lifecycle.py), and Remote-Session-Expires is already present in the 204 response headers.
• /auth/api/validate is the only renewal write (throttled to ≥5 min spacing); it can set a fresh cookie.
• Login/register/remote-auth run over WebSockets with state in process RAM (PoW challenges, remoteauth.instance, authcode 60 s exchange codes). These cannot be cached or replicated.
• The sync feed carries no usable secrets: sessions are keyed by hash_secret output, credentials carry public keys only, and the OIDC signing key is never exported. Knowing a stored session hash does not let you authenticate.
• Restricted paths (/auth/api/admin/, /auth/api/user/, /auth/ws/) are already refused off a domain's own auth host (fastapi/auth_host.py:is_restricted_path), and UI paths already redirect to the auth host. With an auth host configured for a remote domain, the satellite never serves profile, admin or WS traffic for it — browsers and WebSockets go directly to the remote auth host (the WS dispatch rules in fastapi/dispatch.py already allow an origin domain's own auth host).

Option A — caching HTTP reverse proxy

A thin proxy that caches /auth/api/forward, /check, /user-info, /settings responses keyed by (Host, cookie, query) with TTL = min(configured TTL, Remote-Session-Expires − now). Everything else is forwarded verbatim; WebSockets are tunneled; /logout and session-scoped mutations are intercepted to evict cache entries. Optionally stale-while-revalidate so remote-side changes propagate within a TTL without adding latency to any request.

Option B is A plus a small remote addition: a WebSocket change feed (session deleted, user/role/permission changed) so the proxy evicts within one network RTT instead of waiting for the TTL.

• Remote changes: none for A; one additive, RAM-only endpoint for B.
• Staleness: TTL-bounded for anything that bypasses the proxy (admin edits, logout-all from another device) in A; ~1 RTT in B.
• The proxy needs no credentials of its own — every cacheable request is authenticated by the end user's cookie, forwarded on a miss.

Option D — satellite instance with per-domain RAM replica (chosen direction)

The same paskia codebase, with two kinds of domains in the existing multi-domain config:

• Local domains — exactly as today, backed by the local Kanta DB.
• Remote domains — DomainConfig gains remote: {url, token, cache_ttl, refresh_interval}. This is the only new persisted state, and it lives in the satellite's DB. The remote's DB gets nothing new. A remote domain must have an auth host configured (the remote's); this is what makes the simplifications below possible.

For a remote domain the satellite holds a RAM-only read replica of the remote's tables (users, orgs, roles, permissions, credentials, sessions) — another plain DB struct instance, never opened by Kanta, rebuilt from the remote on startup and kept fresh by the sync channel plus snooping on proxied traffic.

No db.data() magic: explicit per-request stores

The first draft proposed making the global db.data() accessor return a replica based on the current_domain contextvar. Rejected: a global accessor whose meaning shifts under the caller is exactly the kind of magic that breeds bugs in no-request-context code (background jobs, OIDC notify, CLI). Instead:

• The runtime Domain object (domains.py) gets a store attribute: a DB instance. For local domains it is the global operations._db; for remote domains it is the replica. Registry build / remote-manager startup attaches it.
• Endpoints resolve the store once, explicitly, from the request: request.state.domain.store (dispatch already puts the domain there), and pass it down as an ordinary parameter. No global accessor changes meaning; local domains observe zero functional change.

Refactor needed to make the hot path store-explicit (small, all in the "manageable core"):

• DB.session_ctx() (db/structs.py) currently leans on struct convenience properties (s.user, user.role, role.org, s.credential, org.permissions) that internally call the global db.data(). Rewrite it to index self.* tables directly. This is a pure correctness fix — it makes session_ctx honest about which DB it reads.
• authsession.session_ctx, permutil.session_context, authz.verify, sessionutil.check_session_age gain an explicit store: DB parameter (or are called with one) instead of reaching the global.
• userinfo.build_user_info takes a store and lists store.sessions/store.credentials filtered by user instead of the user.sessions / user.credentials convenience properties.
• userinfo.build_session_context, api._remote_headers, authz.auth_error_content already operate on the SessionContext alone — no change.
• Struct convenience properties (User.sessions, Session.user, …) keep using the global db.data(); they are only used by local-domain code (admin, profile, CLI) and by Kanta maintenance. Remote-domain request paths simply never call them.

Endpoint categorization for remote domains

1. True special handling (remote-aware logic):

Endpoint	Handling
POST /auth/api/validate	Read from replica; apply the throttled refresh to the replica and write it behind as session_refresh{key, validated, ip, ua} over the sync WS; set the renewed cookie locally (same secret, same rules as today)
POST /auth/api/logout	Forward to remote with the user's cookie; evict the session from the replica immediately (optimistic); the sync event confirms
POST /auth/api/set-session	Forward code redemption to the remote (the 60 s exchange code lives in its RAM); relay the Set-Cookie; the new session arrives via sync event (created during the login ceremony)
GET /auth/api/token-info	Proxy to remote (reset tokens exist only there)
/auth/oidc/token, /userinfo, /keys, /backchannel-logout	Proxy to remote (signing key and OIDC sessions stay there); /keys is cacheable
Avatar GET .../profile.webp	Proxy to remote (storage may be remote-local); not latency-critical

2. Store-explicit, no functional change (the "alike db magic" group — they read the database but only through the store they are handed):

• GET /auth/api/forward — the hot path, pure read from store.
• POST /auth/api/validate — the read/verify half (category 1 covers the refresh half).
• GET /auth/api/check — reads users/roles/orgs/permissions from store.
• GET /auth/api/user-info — build_user_info(store, …).

3. Untouched — no knowledge of remoting:

• GET /auth/api/settings — domain config only; already reports auth_host/auth_site_url, which is how frontends learn to open WebSockets on the remote auth host directly.
• /.well-known/openid-configuration, /.well-known/webauthn — derived from Host and local domain config.
• Frontend statics, /auth/restricted/iframe HTML, reset-token pages — static files; their API calls land in categories 1–2.
• auth_host.redirect_middleware — already redirects UI to the remote auth host and 404s restricted paths off it; with auth host mandatory this is exactly the desired behavior, unchanged.
• The entire admin app, user-profile app, /auth/ws/*, /auth/remote-auth/* — reachable only on the domain's auth host, which for remote domains is the remote itself; the satellite never executes this code for remote domains.
• Dispatch middleware, background cleanup of the local DB, CLI, bootstrap.
• All local-domain handling of every endpoint.

The sync channel (the one remote addition, RAM-only)

New endpoint on the remote, e.g. wss://…/auth/api/sync/ws, plus an emission hook in db/operations.py (publish after each commit; no-op when no peer is connected):

• Remote → satellite: hello{generation, protocol_version} → full snapshot (chunked msgspec; these tables are tiny — 10k sessions ≈ 2 MB) → then event{seq, op: upsert|delete, table, key, payload}. The remote keeps a bounded event ring buffer in RAM; on reconnect the satellite sends its last seq and either gets a replay or a fresh snapshot.
• Satellite → remote: session_refresh write-behind; optionally session_deleted as belt-and-braces alongside proxied logout.
• Authentication: the satellite presents a token the remote reads from env/CLI (e.g. PASKIA_SYNC_TOKENS) — nothing lands in the remote's database. mTLS client certs are an alternative.

Freshness hierarchy, worst case last:

1. Changes made through the satellite: immediate (optimistic eviction on logout, snooped Set-Cookie/login results).
2. Changes made directly on the remote (admin on another device, logout-all): WS event, ~1 network RTT.
3. WS down: per-entry TTL lazy revalidation + periodic full snapshot; on reconnect, replay or re-sync. Fail-open (the replica enforces validated + EXPIRES itself, with its own little sweeper mirroring cleanup_expired) vs fail-closed is configurable per domain.

Deliberately out of scope:

• Running WebAuthn ceremonies on the satellite — duplicates the security-critical verification path and breaks centralized sign-count clone detection; browsers talk to the remote auth host directly anyway.
• OIDC signing on the satellite — key material never leaves the remote.
• Admin/profile functionality for remote domains — deferred to the remote auth host; this is what keeps the satellite's surface small.

D vs. A(+B): what the read-only local state buys

Semantics, not just speed:

• Full SessionContext locally. A answers the exact byte-response it once saw; D computes the answer. New perm/max_age/public query combinations never seen before are served locally by D, while A must go upstream on every cache-key miss. D caches the domain model, so derived answers (effective permissions per host, max_age against credential.last_used, per-request Remote-* header composition) are correct without having been witnessed before.
• One invalidation model. In A, invalidation rules are hand-built per endpoint (query-key mapping, cookie re-keying on renew, 401 variants, /user-info vs /forward overlap). In D, events mutate the replica (upsert/delete by table+key) and every endpoint becomes consistent at once — including endpoints added in the future.
• Degradation behaves like a real instance. With the remote down, D keeps serving a coherent auth service from the replica (expiry enforced locally). A serves a pile of unrelated cached responses with gaps wherever the cache was cold.
• Multi-domain uniformity. D is a property of a domain in the existing registry: local rp-ids and remote rp-ids coexist, and the same instance genuinely is the auth server for its local domains. A is a separate component bolted in front of specific URLs.
• User simplicity. D is configured once as domain config (remote.url + token); A needs deployment and key-mapping discipline per frontend application.

What D costs

• A store-explicitness refactor in the core. DB.session_ctx must stop leaning on global-accessing convenience properties; the verify/userinfo helper chain gains a store parameter. Small and mechanical, but it touches the security-critical read path, so it wants careful review and tests proving local-domain behavior is byte-identical. (The rejected alternative — a contextvar-dependent db.data() — avoided this refactor at the price of a global whose meaning shifts under the caller.)
• A sync protocol to design and version. Snapshot + sequenced events + ring-buffer replay + reconnect reconciliation, plus a protocol-version handshake because remote and satellite releases can drift in SessionContext semantics. A needs no protocol at all.
• A trusted satellite host. The replica holds everything the remote DB has except the OIDC key (public keys, hashed session keys, user data) in RAM. The host must be as trusted as the remote. A caches only what traffic it saw, in the shape it saw it.
• Remote code, albeit additive. D requires the sync endpoint and commit hooks on the remote (RAM-only, inert when unused, token-gated). A requires zero remote changes; B requires the same endpoint in a thinner form.
• Replica housekeeping — its own expiry sweeper, write-behind ordering, optimistic eviction vs. event confirmation. A's failure modes are dumber and narrower.

Summary

A (+B) is the right tool when the goal is "make forward-auth fast in front of an untouched server" — small, deployable anywhere, byte-compatible. D is the right tool when the satellite should be a paskia instance for its remote domains: one consistency model, correct answers for un-cached query shapes, graceful degradation, and per-domain mixing with local rp-ids. With the auth host made mandatory for remote domains, D's surface shrinks to four locally served read endpoints, a handful of proxied mutations, an explicit store parameter in the core read path, and one RAM-only sync endpoint on the remote.