Tidying up log, correct HTTP status for error pages.

- Resolve threading deadlock when multiple watch request arrived at the same moment.
- Implement more graceful server exit.
- Reduce excessive logging.
- Fix unix socket clearing; until Sanic starts accepting Path for unix socket name.

README.md

@@ -1,8 +1,8 @@
 # Cista Web Storage
 
-<img src="https://git.zi.fi/Vasanko/cista-storage/raw/branch/main/docs/cista.jpg" align=right width=250>
+<img src="https://git.zi.fi/Vasanko/cista-storage/raw/branch/main/docs/cista.webp" align=left width=250>
 
-Cista takes its name from the ancient cistae, metal containers used by Greeks and Egyptians to safeguard valuable items. This modern application provides a browser interface for secure and accessible file storage, echoing the trust and reliability of its historical namesake.
+Cista takes its name from the ancient *cistae*, metal containers used by Greeks and Egyptians to safeguard valuable items. This modern application provides a browser interface for secure and accessible file storage, echoing the trust and reliability of its historical namesake.
 
 This is a cutting-edge **file and document server** designed for speed, efficiency, and unparalleled ease of use. Experience **lightning-fast browsing**, thanks to the file list maintained directly in your browser and updated from server filesystem events, coupled with our highly optimized code. Fully **keyboard-navigable** and with a responsive layout, Cista flawlessly adapts to your devices, providing a seamless experience wherever you are. Our powerful **instant search** means you're always just a few keystrokes away from finding exactly what you need. Press **1/2/3** to switch ordering, navigate with all four arrow keys (+Shift to select). Or click your way around on **breadcrumbs that remember where you were**.
 
@@ -46,7 +46,9 @@ To use your own TLS certificates, place them in the config folder and run:
 cista -l cista.example.com
 ```
 
-Most admins instead find the [Caddy](https://caddyserver.com/) web server convenient for its auto TLS certificates and all. A proxy also allows running multiple web services or Cista instances on the same IP address. Caddy configuration **/etc/caddy/Caddyfile** is dead simple:
+Most admins instead find the [Caddy](https://caddyserver.com/) web server convenient for its auto TLS certificates and all. A proxy also allows running multiple web services or Cista instances on the same IP address but different (sub)domains.
+
+`/etc/caddy/Caddyfile`:
 
 ```Caddyfile
 cista.example.com {
@@ -72,7 +74,7 @@ pip install -e '.[dev]'
 cista --dev -l :8000 /path/to/files
 ```
 
-We use `hatch shell` for installing on a virtual environment, to not disturb the rest of the system with our hacking.
+We use `hatch shell` for installing on a virtual environment, to avoid disturbing the rest of the system with our hacking.
 
 Vue is used to build files in `cista/wwwroot`, included prebuilt in the Python package. Running `hatch build` builds the frontend and creates a NodeJS-independent Python package.
 
@@ -80,9 +82,9 @@ Vue is used to build files in `cista/wwwroot`, included prebuilt in the Python p
 
 This setup allows easy addition of storages, each with its own domain, configuration, and files.
 
-Assuming a restricted user account **storage** for serving files and that cista is installed system-wide or on this account (check with `sudo -u storage -s`). Alternatively, use `pipx run cista` or `hatch run cista` as the ExecStart command.
+Assuming a restricted user account `storage` for serving files and that cista is installed system-wide or on this account (check with `sudo -u storage -s`). Alternatively, use `pipx run cista` or `hatch run cista` as the ExecStart command.
 
-Create **/etc/systemd/system/cista@.service**:
+Create `/etc/systemd/system/cista@.service`:
 
 ```ini
 [Unit]
@@ -90,7 +92,7 @@ Description=Cista storage %i
 
 [Service]
 User=storage
-ExecStart=cista -c /srv/cista/%i -l /srv/cista/%i/socket /media/storage/@%i/
+ExecStart=cista -c /srv/cista/%i -l /srv/cista/%i/socket /media/storage/%i
 Restart=always
 
 [Install]
@@ -105,9 +107,9 @@ systemctl enable --now cista@foo.example.com
 systemctl enable --now cista@bar.example.com
 ```
 
-Public exposure is easiest using the Caddy web server, but Nginx or others also work. Run the server with -l domain.example.com if you have TLS certificates in the config folder.
+Public exposure is easiest using the Caddy web server.
 
-**/etc/caddy/Caddyfile**:
+`/etc/caddy/Caddyfile`:
 
 ```Caddyfile
 foo.example.com, bar.example.com {

cista/api.py

@@ -111,13 +111,24 @@ async def watch(req, ws):
     )
     uuid = token_bytes(16)
     try:
-        with watching.state.lock:
-            q = watching.pubsub[uuid] = asyncio.Queue()
-            # Init with disk usage and full tree
-            await ws.send(watching.format_space(watching.state.space))
-            await ws.send(watching.format_root(watching.state.root))
+        q, space, root = await asyncio.get_event_loop().run_in_executor(
+            req.app.ctx.threadexec, subscribe, uuid, ws
+        )
+        await ws.send(space)
+        await ws.send(root)
         # Send updates
         while True:
             await ws.send(await q.get())
     finally:
         del watching.pubsub[uuid]
+
+
+def subscribe(uuid, ws):
+    with watching.state.lock:
+        q = watching.pubsub[uuid] = asyncio.Queue()
+        # Init with disk usage and full tree
+        return (
+            q,
+            watching.format_space(watching.state.space),
+            watching.format_root(watching.state.root),
+        )

cista/app.py

@@ -1,6 +1,7 @@
 import asyncio
 import datetime
 import mimetypes
+import threading
 from concurrent.futures import ThreadPoolExecutor
 from pathlib import Path, PurePath, PurePosixPath
 from stat import S_IFDIR, S_IFREG
@@ -12,7 +13,7 @@ import sanic.helpers
 from blake3 import blake3
 from sanic import Blueprint, Sanic, empty, raw
 from sanic.exceptions import Forbidden, NotFound
-from sanic.log import logging
+from sanic.log import logger
 from stream_zip import ZIP_AUTO, stream_zip
 
 from cista import auth, config, session, watching
@@ -31,14 +32,15 @@ app.exception(Exception)(handle_sanic_exception)
 @app.before_server_start
 async def main_start(app, loop):
     config.load_config()
-    await watching.start(app, loop)
     app.ctx.threadexec = ThreadPoolExecutor(
         max_workers=8, thread_name_prefix="cista-ioworker"
     )
+    await watching.start(app, loop)
 
 
 @app.after_server_stop
 async def main_stop(app, loop):
+    quit.set()
     await watching.stop(app, loop)
     app.ctx.threadexec.shutdown()
 
@@ -122,7 +124,7 @@ def _load_wwwroot(www):
     if not wwwnew:
         msg = f"Web frontend missing from {base}\n  Did you forget: hatch build\n"
         if not www:
-            logging.warning(msg)
+            logger.warning(msg)
         if not app.debug:
             msg = "Web frontend missing. Cista installation is broken.\n"
         wwwnew[""] = (
@@ -141,7 +143,7 @@ def _load_wwwroot(www):
 async def start(app):
     await load_wwwroot(app)
     if app.debug:
-        app.add_task(refresh_wwwroot())
+        app.add_task(refresh_wwwroot(), name="refresh_wwwroot")
 
 
 async def load_wwwroot(app):
@@ -151,9 +153,12 @@ async def load_wwwroot(app):
     )
 
 
+quit = threading.Event()
+
+
 async def refresh_wwwroot():
-    while True:
-        await asyncio.sleep(0.5)
+    try:
+        while not quit.is_set():
             try:
                 wwwold = www
                 await load_wwwroot(app)
@@ -169,9 +174,10 @@ async def refresh_wwwroot():
                 if changes:
                     print(f"Updated wwwroot:\n{changes}", end="", flush=True)
             except Exception as e:
-            print("Error loading wwwroot", e)
-        if not app.debug:
-            return
+                print(f"Error loading wwwroot: {e!r}")
+            await asyncio.sleep(0.5)
+    except asyncio.CancelledError:
+        pass
 
 
 @app.route("/<path:path>", methods=["GET", "HEAD"])
@@ -251,7 +257,7 @@ async def zip_download(req, keys, zipfile, ext):
             for chunk in stream_zip(local_files(files)):
                 asyncio.run_coroutine_threadsafe(queue.put(chunk), loop).result()
         except Exception:
-            logging.exception("Error streaming ZIP")
+            logger.exception("Error streaming ZIP")
             raise
         finally:
             asyncio.run_coroutine_threadsafe(queue.put(None), loop)

cista/auth.py

@@ -71,7 +71,7 @@ def verify(request, *, privileged=False):
             raise Forbidden("Access Forbidden: Only for privileged users", quiet=True)
     elif config.config.public or request.ctx.user:
         return
-    raise Unauthorized("Login required", "cookie", quiet=True)
+    raise Unauthorized(f"Login required for {request.path}", "cookie", quiet=True)
 
 
 bp = Blueprint("auth")

cista/serve.py

@@ -51,7 +51,7 @@ def parse_listen(listen):
             raise ValueError(
                 f"Directory for unix socket does not exist: {unix.parent}/",
             )
-        return "http://localhost", {"unix": unix}
+        return "http://localhost", {"unix": unix.as_posix()}
     if re.fullmatch(r"(\w+(-\w+)*\.)+\w{2,}", listen, re.UNICODE):
         return f"https://{listen}", {"host": listen, "port": 443, "ssl": True}
     try:

cista/util/apphelpers.py

@@ -21,7 +21,6 @@ def jres(data, **kwargs):
 
 
 async def handle_sanic_exception(request, e):
-    logger.exception(e)
     context, code = {}, 500
     message = str(e)
     if isinstance(e, SanicException):
@@ -42,7 +41,9 @@ async def handle_sanic_exception(request, e):
         res.cookies.add_cookie("message", message, max_age=5)
         return res
     # Otherwise use Sanic's default error page
-    return errorpages.HTMLRenderer(request, e, debug=request.app.debug).full()
+    res = errorpages.HTMLRenderer(request, e, debug=request.app.debug).full()
+    res.status = status  # Unfortunately Sanic <23.12 doesn't set this
+    return res
 
 
 def websocket_wrapper(handler):
@@ -54,13 +55,14 @@ def websocket_wrapper(handler):
             auth.verify(request)
             await handler(request, ws, *args, **kwargs)
         except Exception as e:
-            logger.exception(e)
             context, code, message = {}, 500, str(e) or "Internal Server Error"
             if isinstance(e, SanicException):
                 context = e.context or {}
                 code = e.status_code
             message = f"⚠️ {message}" if code < 500 else f"🛑 {message}"
             await asend(ws, ErrorMsg({"code": code, "message": message, **context}))
+            if not getattr(e, "quiet", False):
+                logger.exception(f"{code} {e!r}")
             raise
 
     return wrapper

cista/watching.py

@@ -1,15 +1,16 @@
 import asyncio
 import shutil
-import stat
 import sys
 import threading
 import time
+from contextlib import suppress
 from os import stat_result
 from pathlib import Path, PurePosixPath
+from stat import S_ISDIR
 
 import msgspec
 from natsort import humansorted, natsort_keygen, ns
-from sanic.log import logging
+from sanic.log import logger
 
 from cista import config
 from cista.fileio import fuid
@@ -113,96 +114,17 @@ class State:
 
 state = State()
 rootpath: Path = None  # type: ignore
-quit = False
-modified_flags = (
-    "IN_CREATE",
-    "IN_DELETE",
-    "IN_DELETE_SELF",
-    "IN_MODIFY",
-    "IN_MOVE_SELF",
-    "IN_MOVED_FROM",
-    "IN_MOVED_TO",
-)
+quit = threading.Event()
+
+## Filesystem scanning
 
 
-def watcher_thread(loop):
-    global rootpath
-    import inotify.adapters
-
-    while not quit:
-        rootpath = config.config.path
-        i = inotify.adapters.InotifyTree(rootpath.as_posix())
-        # Initialize the tree from filesystem
-        new = walk()
-        with state.lock:
-            old = state.root
-            if old != new:
-                state.root = new
-                broadcast(format_update(old, new), loop)
-
-        # The watching is not entirely reliable, so do a full refresh every 30 seconds
-        refreshdl = time.monotonic() + 30.0
-
-        for event in i.event_gen():
-            if quit:
-                return
-            # Disk usage update
-            du = shutil.disk_usage(rootpath)
-            space = Space(*du, storage=state.root[0].size)
-            if space != state.space:
-                state.space = space
-                broadcast(format_space(space), loop)
-                break
-            # Do a full refresh?
-            if time.monotonic() > refreshdl:
-                break
-            if event is None:
-                continue
-            _, flags, path, filename = event
-            if not any(f in modified_flags for f in flags):
-                continue
-            # Update modified path
-            path = PurePosixPath(path) / filename
-            try:
-                update(path.relative_to(rootpath), loop)
-            except Exception as e:
-                print("Watching error", e, path, rootpath)
-                raise
-        i = None  # Free the inotify object
-
-
-def watcher_thread_poll(loop):
-    global rootpath
-
-    while not quit:
-        rootpath = config.config.path
-        new = walk()
-        with state.lock:
-            old = state.root
-            if old != new:
-                state.root = new
-                broadcast(format_update(old, new), loop)
-
-        # Disk usage update
-        du = shutil.disk_usage(rootpath)
-        space = Space(*du, storage=state.root[0].size)
-        if space != state.space:
-            state.space = space
-            broadcast(format_space(space), loop)
-
-        time.sleep(2.0)
-
-
-def walk(rel=PurePosixPath()) -> list[FileEntry]:  # noqa: B008
+def walk(rel: PurePosixPath, stat: stat_result | None = None) -> list[FileEntry]:
     path = rootpath / rel
+    ret = []
     try:
-        st = path.stat()
-    except OSError:
-        return []
-    return _walk(rel, int(not stat.S_ISDIR(st.st_mode)), st)
-
-
-def _walk(rel: PurePosixPath, isfile: int, st: stat_result) -> list[FileEntry]:
+        st = stat or path.stat()
+        isfile = not S_ISDIR(st.st_mode)
         entry = FileEntry(
             level=len(rel.parts),
             name=rel.name,
@@ -213,36 +135,43 @@ def _walk(rel: PurePosixPath, isfile: int, st: stat_result) -> list[FileEntry]:
         )
         if isfile:
             return [entry]
+        # Walk all entries of the directory
         ret = [entry]
-    path = rootpath / rel
-    try:
         li = []
         for f in path.iterdir():
-            if quit:
+            if quit.is_set():
                 raise SystemExit("quit")
             if f.name.startswith("."):
                 continue  # No dotfiles
+            with suppress(FileNotFoundError):
                 s = f.stat()
-            li.append((int(not stat.S_ISDIR(s.st_mode)), f.name, s))
-        for [isfile, name, s] in humansorted(li):
-            if quit:
-                raise SystemExit("quit")
-            subtree = _walk(rel / name, isfile, s)
-            child = subtree[0]
+                li.append((int(not S_ISDIR(s.st_mode)), f.name, s))
+        # Build the tree as a list of FileEntries
+        for [_, name, s] in humansorted(li):
+            sub = walk(rel / name, stat=s)
+            ret.extend(sub)
+            child = sub[0]
             entry.mtime = max(entry.mtime, child.mtime)
             entry.size += child.size
-            ret.extend(subtree)
     except FileNotFoundError:
         pass  # Things may be rapidly in motion
-    except OSError as e:
-        print("OS error walking path", path, e)
+    except OSError:
+        logger.error(f"Watching {path=}: {e!r}")
     return ret
 
 
-def update(relpath: PurePosixPath, loop):
-    """Called by inotify updates, check the filesystem and broadcast any changes."""
-    if rootpath is None or relpath is None:
-        print("ERROR", rootpath, relpath)
+def update_root(loop):
+    """Full filesystem scan"""
+    new = walk(PurePosixPath())
+    with state.lock:
+        old = state.root
+        if old != new:
+            state.root = new
+            broadcast(format_update(old, new), loop)
+
+
+def update_path(relpath: PurePosixPath, loop):
+    """Called on FS updates, check the filesystem and broadcast any changes."""
     new = walk(relpath)
     with state.lock:
         old = state[relpath]
@@ -256,6 +185,22 @@ def update(relpath: PurePosixPath, loop):
         broadcast(format_update(old, state.root), loop)
 
 
+def update_space(loop):
+    """Called periodically to update the disk usage."""
+    du = shutil.disk_usage(rootpath)
+    space = Space(*du, storage=state.root[0].size)
+    # Update only on difference above 1 MB
+    tol = 10**6
+    old = msgspec.structs.astuple(state.space)
+    new = msgspec.structs.astuple(space)
+    if any(abs(o - n) > tol for o, n in zip(old, new, strict=True)):
+        state.space = space
+        broadcast(format_space(space), loop)
+
+
+## Messaging
+
+
 def format_update(old, new):
     # Make keep/del/insert diff until one of the lists ends
     oidx, nidx = 0, 0
@@ -316,20 +261,73 @@ async def abroadcast(msg):
             queue.put_nowait(msg)
     except Exception:
         # Log because asyncio would silently eat the error
-        logging.exception("Broadcast error")
+        logger.exception("Broadcast error")
+
+
+## Watcher thread
+
+
+def watcher_inotify(loop):
+    """Inotify watcher thread (Linux only)"""
+    import inotify.adapters
+
+    modified_flags = (
+        "IN_CREATE",
+        "IN_DELETE",
+        "IN_DELETE_SELF",
+        "IN_MODIFY",
+        "IN_MOVE_SELF",
+        "IN_MOVED_FROM",
+        "IN_MOVED_TO",
+    )
+    while not quit.is_set():
+        i = inotify.adapters.InotifyTree(rootpath.as_posix())
+        # Initialize the tree from filesystem
+        update_root(loop)
+        trefresh = time.monotonic() + 30.0
+        tspace = time.monotonic() + 5.0
+        # Watch for changes (frequent wakeups needed for quiting)
+        for event in i.event_gen(timeout_s=0.1):
+            if quit.is_set():
+                break
+            t = time.monotonic()
+            # The watching is not entirely reliable, so do a full refresh every 30 seconds
+            if t >= trefresh:
+                break
+            # Disk usage update
+            if t >= tspace:
+                tspace = time.monotonic() + 5.0
+                update_space(loop)
+            # Inotify event, update the tree
+            if event and any(f in modified_flags for f in event[1]):
+                # Update modified path
+                update_path(PurePosixPath(event[2]) / event[3], loop)
+
+        del i  # Free the inotify object
+
+
+def watcher_poll(loop):
+    """Polling version of the watcher thread."""
+    while not quit.is_set():
+        update_root(loop)
+        update_space(loop)
+        quit.wait(2.0)
 
 
 async def start(app, loop):
+    global rootpath
     config.load_config()
+    rootpath = config.config.path
     use_inotify = sys.platform == "linux"
     app.ctx.watcher = threading.Thread(
-        target=watcher_thread if use_inotify else watcher_thread_poll,
+        target=watcher_inotify if use_inotify else watcher_poll,
         args=[loop],
+        # Descriptive name for system monitoring
+        name=f"cista-watcher {rootpath}",
     )
     app.ctx.watcher.start()
 
 
 async def stop(app, loop):
-    global quit
-    quit = True
+    quit.set()
     app.ctx.watcher.join()

frontend/README.md

@@ -1,6 +1,6 @@
-# cista-front
+# Cista Vue Frontend
 
-This template should help get you started developing with Vue 3 in Vite.
+The frontend is a Single-Page App implemented with Vue 3. Development uses the Vite server together with the main Python backend, but in production the latter also serves the prebuilt frontend files.
 
 ## Recommended IDE Setup
 
@@ -17,24 +17,30 @@ If the standalone TypeScript plugin doesn't feel fast enough to you, Volar has a
     2) Find `TypeScript and JavaScript Language Features`, right click and select `Disable (Workspace)`
 2. Reload the VSCode window by running `Developer: Reload Window` from the command palette.
 
-## Customize configuration
+## Hot-Reload for Development
 
-See [Vite Configuration Reference](https://vitejs.dev/config/).
+### Run the backend
 
-## Project Setup
-
-```sh
-npm install
+```fish
+hatch shell
+cista --dev -l :8000
 ```
 
-### Compile and Hot-Reload for Development
+### And the Vite server (in another terminal)
 
-```sh
+```fish
+cd frontend
+npm install
 npm run dev
 ```
+Browse to Vite, which will proxy API requests to port 8000. Both servers live reload changes.
+
 
 ### Type-Check, Compile and Minify for Production
 
-```sh
+This is also called by `hatch build` during Python packaging:
+
+```fish
 npm run build
 ```
+