A corrupt TIFF in production produced a wall of ffmpeg error output:
pyvips could not decode it, the generic ffmpeg fallback was tried, and
ffmpeg's TIFF decoder failed just the same — with banner, configuration
and stream-mapping spam included.
- Non-HEIC images are now decoded by pyvips only; a pyvips failure
raises a clean one-line ValueError ("cannot decode image: ...", a 422
like any other undecodable file) instead of invoking ffmpeg. The
ffmpeg path is kept for HEIC/HEIF, where pyvips genuinely falls short
(tile assembly, HDR metadata).
- ffmpeg runs with -hide_banner -loglevel error -nostats: error output
is still shown on failure, without the version/configuration/progress
noise. The -s insertion index no longer depends on fixed positions.
- AVIF saves: replace deprecated strip=True with keep="none" (libvips
8.15+; production already runs a version that deprecates strip).
- Set the pyvips logger to WARNING in the worker: its INFO messages
("threadpool completed ...") are pure spam on every operation.
Worker stderr is now inherited by the parent, so its log lines land in the
server log — but they arrived with the default logging format and a noisy
"preview-worker config=..." line at every spawn.
- Extract the emoji level-prefix formatting from cista.sanic_logging into
cista.util.logformat, which has no Sanic dependency (the worker must not
import Sanic: import-time prints could corrupt the stdout protocol).
- Worker configures its stderr handler with the same emoji prefixes plus a
worker[pid] tag, and the config-loaded info message is removed.
Production symptom: previews of all types (pdf/pyvips/onlyoffice) start
hitting the 10s timeout and never recover until server restart, while the
rest of the server stays healthy.
Root cause (reproduced on Python 3.12): workers were spawned with
stderr=PIPE that nothing drained after startup. Once the OS pipe buffer
filled from accumulated worker tracebacks and library warnings, asyncio
flow control stopped the parent reading it and the worker blocked forever
mid-request on a stderr write. The 10s timeout then fired, but
_replace_worker hung forever in proc.wait() even after kill() — the
flow-control-paused pipe transport never sees EOF — permanently wedging
one dispatcher per stuck worker. Once all dispatchers were stuck, every
preview request timed out. Restart cleared it.
Fixes:
- Spawn workers with inherited stderr (stderr=None) so worker diagnostics
go straight to the server log and no undrained pipe can exist.
- Bound proc.wait() in worker kill() with a 5s grace timeout so a wedged
transport can never hang a dispatcher; log the worker pid instead.
- Guard the dispatch loop with an outer exception handler so a dispatcher
can never die silently and shrink pool capacity.
- Retry failed worker replacement spawns with 1s-30s backoff instead of
silently shrinking the pool.
- Fix latent crash: except-tuple referenced msgspec.json.DecodeError,
which does not exist in the installed msgspec; any protocol failure
would itself raise AttributeError. Use msgspec.DecodeError.
- Worker: redirect Python-level sys.stdout to stderr in persistent mode
and keep the raw buffer solely for the binary protocol, so a library
print() can never corrupt the command channel again (cf. the pymupdf
deprecation warning that crashed workers at startup).
- Worker: close the pymupdf document explicitly in process_pdf.
- Log worker pid on timeout/protocol/checksum failures, and log failed
kills and replacement retries, for future production diagnostics.
Add tests/test_preview_pool.py with an end-to-end regression recreating
the wedged-worker setup (piped, undrained stderr) plus kill-grace,
respawn-retry and dispatcher-survival tests.
Newer ruff flagged CPY001 across the codebase (copyright notices are not
wanted here, rule disabled) and PLR0917 on format_access_log. duration_ms
and extra are now keyword-only at the single call site.
The ffmpeg fallback in the preview worker inherited the worker's stdin
pipe (the framed request protocol). When a slow conversion was killed
at the 10s timeout, the orphaned ffmpeg grandchild kept that pipe open,
so the parent's proc.wait() blocked forever waiting for pipe EOF —
permanently sticking one dispatcher per event until the whole pool
starved and every preview request (pdf, image, office) returned 503.
- Run ffmpeg with stdin=DEVNULL (also stops it eating protocol bytes)
- Drop start_new_session (only needed for group kills, POSIX-only)
- Stop logging the master secret at worker startup
- Add cista/util/diskspace.py with MIN_FREE_BYTES limit and cached
check_free_space() helper.
- Check available space in File.write() before ftruncate/write.
- Catch ENOSPC in File.write and re-raise as InsufficientStorageError.
- upload_file_chunk catches both proactive and ENOSPC errors and
returns HTTP 507 Insufficient Storage.
- Add tests for low-disk rejection and ENOSPC handling.
_get_image_dimensions now relies solely on pyvips header reading.
If pyvips cannot read the header the function returns None and the
caller encodes at full resolution rather than falling back to ffprobe.
Remove unused stdlib json import.
- Use ffmpeg CLI directly for HEIC/HEIF previews (bypassing pyvips which
cannot encode >8-bit AVIF on current system). ffmpeg handles tile
assembly, EXIF rotation, HDR metadata and ICC profiles automatically.
- pyvips remains primary for other image formats; ffmpeg is fallback.
- Simplify _get_image_dimensions() to use pyvips header read for all
formats including HEIC, with ffprobe as fallback.
- Move all imports to top of preview.py; remove lazy OnlyOffice import.
- Worker pool: add readiness handshake (\x01 byte), bounded idle wait
with PREVIEW_TIMEOUT, eager module import before signalling.
- Fix dispatch data check bug (was 'if data is not None' -> 'if data').
- Add logger.exception for unhandled preview/dispatch errors.
- EmojiFormatter now includes tracebacks for exception logs.
- Return 503 on preview timeouts instead of 504.
- Move OnlyOffice conversion out of worker subprocesses into async event loop
using httpx.AsyncClient with shared client and clean shutdown hook
- Add OOConversionManager with in-flight deduplication (asyncio.Future) and
configurable concurrency limit (OO_MAX_CONCURRENT=2)
- Add 10s total timeout for office previews via asyncio.wait_for
- Return 503 when OnlyOffice is unavailable, 504 on timeout
- Remove office handling from worker dispatch(); workers now only do
images, video, and PDFs
- Replace PNG tempfile bridge with framed binary input protocol on worker
stdin: (json_size)(data_size)(json)(raw_data)
- Add process_image_buffer() for in-memory AVIF conversion via pyvips
- Add office_previews to WS server message, cached every 30s from OO
availability check
- Frontend gates only office document previews on office_previews flag;
images, video, PDFs remain unconditional
- Change default OnlyOffice port from 8080 to 8988
Replace Aspose.Words with OnlyOffice Document Server for generating
bitmap previews of office documents (Word, Excel, PowerPoint, etc.).
Backend:
- Add cista/onlyoffice.py conversion client
- Convert office docs directly to PNG via OnlyOffice, then AVIF via pyvips
- Make office previews optional based on OnlyOffice availability
- Remove Aspose.Words dependency and all related code
- Add spreadsheet and presentation format support
Frontend:
- Mark office files as previewable in Document.ts
- Add office extensions to MediaPreview.vue preview list
- Fix pre-existing @ts-ignore in HeaderMain.vue
Tests:
- Fix test_lrucache.py parameter name (open -> opener)
Also run ruff format across the codebase to satisfy linter checks.
Implement complete WebDAV file serving compatible with various clients from Windows File Explorer to more specialized sync tools. The old control WebSocket has been updated to part-DAV, part REST API instead. Implemented user:pass BASIC auth. Added UI and backend for creating tokens that avoid the need to use actual username and password for requests from CLI or DAV.
Increases block size to 16 MiB and adjusts progress display to work smoothly with that. Also removes download WS that was already unused. Provides faster upload speed than over WS.
LeoVasanko