Fix scanner rescan bugs and wasted work
Bugs fixed: - Partial rescans no longer replace whole index entries: Upsert events now carry the scanned torrent paths and IndexStore merges partial rebuilds, so touching one season no longer drops the others from listings. - Deleted torrents are now detected: discovery reports the full candidate set via a new Sync event after each completed scan, and the store prunes file entries/episodes/seasons/items whose torrents vanished. - Seen mtimes are committed only after a scan completes successfully, so cancelled/failed scans retry their items. - Showreel worker no longer rebroadcasts stale whole items; it sends narrow MovieShowreel/EpisodeReel events that update only reel fields. - Permanently failing reel generations (e.g. DoVi/libplacebo on GPU-less machines) and short-video re-queueing are no longer retried every scan: reel outcomes persist in reel-state.json with exponential backoff. Optimizations: - Persist scan state (scan-state.json) and ffmpeg probe results (probe-cache.json, keyed by mtime+size, failures included) under .mediahive/, written only when changed. A warm restart over an unchanged library drops from ~57 s + 467 queued reel tasks to ~0.2 s with an empty queue (measured on a 163-item root). - Bounded parallelism (semaphores + gather) for cast-profile downloads, TMDb title lookups, and season-detail fetches. - Incremental TMDb-id index bookkeeping instead of rebuilding per upsert; removed dead trigger_scan. See docs/scanning-review.md for the full findings/fixes/measurements report.
This commit is contained in:
@@ -0,0 +1,352 @@
|
||||
# Scanner review — findings, fixes, and measured results
|
||||
|
||||
Date: 2026-09-03 (review); fixes implemented same day.
|
||||
Scope: `mediahive/hivescan/*`, `mediahive/index_store.py`, `mediahive/root_registry.py`,
|
||||
`mediahive/models/events.py`
|
||||
Method: code review plus instrumented runs of the real `RootScanner` (monkeypatched
|
||||
timers around every ffmpeg invocation, HTTP request, and filesystem primitive).
|
||||
|
||||
Structure of this document:
|
||||
|
||||
- **Section 1** records the findings as measured against the *pre-fix* code
|
||||
(line references are from that revision).
|
||||
- **Section 2** describes the fixes that were implemented for each finding.
|
||||
- **Section 3** gives before/after measurements.
|
||||
- **Appendix A** (blob storage options) is kept for reference only; it was
|
||||
explicitly decided **not** to change the on-disk storage format for now.
|
||||
|
||||
Test environment (details omitted intentionally): the media library lives on a
|
||||
network-mounted filesystem (SMB/CIFS). Two roots were measured:
|
||||
|
||||
- **Subset root**: 163 torrents (121 movies, 40 series entries), stale/empty index.
|
||||
- **Full library root**: existing index with 1127 movies, 159 series,
|
||||
1608 episodes, ~2983 video files, ~35k people records.
|
||||
|
||||
Environment characteristic that dominates several measurements:
|
||||
`stat`/`scandir`/`exists` on the network mount are ~0.1 ms (attribute caching
|
||||
works), but **every small file write costs ~0.2 s** (synchronous write-through).
|
||||
The scanner writes thousands of small files into `.mediahive/` on that mount.
|
||||
|
||||
---
|
||||
|
||||
## 1. Findings (pre-fix)
|
||||
|
||||
### F1 — Partial rescan replaces whole entries (the "disappearing seasons" bug)
|
||||
|
||||
**This is a correctness bug, not a performance issue.**
|
||||
|
||||
Chain of events:
|
||||
|
||||
1. Discovery mtime-gates per torrent directory/file. Touching one season
|
||||
directory of a series yields a `downloads` list containing *only that
|
||||
season*.
|
||||
2. `_process_series` rebuilds the `Series` object from only the items it was
|
||||
given.
|
||||
3. `IndexStore.upsert_series` **replaced the entire entry** and broadcast the
|
||||
partial series to all connected clients.
|
||||
|
||||
Measured end-to-end on a 5-season series (5 separate season torrents,
|
||||
110 episodes):
|
||||
|
||||
```
|
||||
full scan -> index entry seasons [1,2,3,4,5] (110 episodes)
|
||||
touch season 4 -> rescan 0.2 s, emits ONE upsert: seasons [4] (22 episodes)
|
||||
-> index entry is now seasons [4] — seasons 1-3,5 gone
|
||||
```
|
||||
|
||||
The missing seasons returned only when a scan happened to include all seasons
|
||||
again — in practice the next **process restart**, because the seen-mtimes map
|
||||
(F2) was memory-only and forced a full rediscovery at startup.
|
||||
|
||||
Same bug class, other variants:
|
||||
|
||||
- **Movies**: a touched version directory dropped the other versions of the
|
||||
same movie from the listing.
|
||||
- **TMDb dedupe collapse**: when a partial entry arrived under a *different*
|
||||
item id for an already-known TMDb id, the old **complete** entry was
|
||||
explicitly deleted.
|
||||
- **Deletions were never detected**: discovery only ever added to
|
||||
`_seen_mtimes`; nothing emitted removals. A torrent deleted from disk stayed
|
||||
in the index forever.
|
||||
|
||||
### F2 — No persistent scan state: every restart was a full reprocess
|
||||
|
||||
All scanner state was process memory: `_seen_mtimes`, the ffmpeg probe cache,
|
||||
and the episode/playable-file/bluray-probe caches.
|
||||
|
||||
Consequences measured:
|
||||
|
||||
- **Steady-state rescan within one process: 0.2 s** (subset root, nothing
|
||||
changed) — mtime gating worked fine while the process lived.
|
||||
- **Warm-restart scan (fresh process caches, all disk caches warm): 56.5 s**
|
||||
for the same 163 items, of which **52.9 s (94 %) was re-running ffmpeg probes
|
||||
on all 469 video files** (21 s `ffmpeg -i` + 32 s `showinfo` passes on HDR
|
||||
files). TMDb was 100 % disk-cache hits and cost 0.8 s total.
|
||||
- Scaled to the full library: every application restart re-probed **~2983
|
||||
files ≈ 6–8 minutes** of sequential ffmpeg, during which the whole index was
|
||||
re-derived and re-upserted item by item (see F4).
|
||||
|
||||
### F3 — Preview (showreel) generation: restart storms and infinite retries
|
||||
|
||||
- **Every scan that included an item enqueued all of its reel tasks**, whether
|
||||
or not the reels existed. Existence was only checked later by the serial
|
||||
worker. A full scan of the subset root enqueued **467** tasks; the full
|
||||
library would enqueue ~2700.
|
||||
- **Nothing was persisted about the queue.** A restart before the queue drained
|
||||
started everything over.
|
||||
- **Failures were never recorded.** In the drain test, 3 of 18 movies failed
|
||||
deterministically (DoVi profile 7 titles require `libplacebo` tonemapping,
|
||||
which fails on GPU-less machines; one file has a matroska demux error). The
|
||||
same files were retried on every subsequent drain — ~1.5 s of probing plus
|
||||
crop detection plus an error task broadcast to every client, **forever**.
|
||||
- The reel-existence check required **all five** reels; short videos
|
||||
legitimately produce fewer, so they were treated as "missing" and re-queued
|
||||
on every scan, generating nothing new each time.
|
||||
- Measured generation pace with software AV1 encoding: ~16–28 s per movie
|
||||
(5 clips), ~4–5 s per episode clip.
|
||||
- The reel worker rebroadcast **whole items** built from stale scan data,
|
||||
clobbering newer store state.
|
||||
|
||||
### F4 — Degraded operation while a full scan is in progress
|
||||
|
||||
- Items were re-upserted one by one as processed; combined with F1, any
|
||||
partial rescan interleaved with normal use made listings lose data until the
|
||||
next restart.
|
||||
- The showreel worker broadcast an **error task for every permanent failure on
|
||||
every scan** (F3), producing user-visible noise.
|
||||
- The index snapshot was rewritten every 5 s while dirty; with 1286+ items
|
||||
that is a ~7.5 MB serialize + write per flush, continuously, for the
|
||||
duration of a scan.
|
||||
- `upsert_*` rebuilt the TMDb-id lookup maps on **every** upsert — O(n²) per
|
||||
scan. Measured negligible; fixed anyway as part of the merge work.
|
||||
|
||||
### F5 — Cold-scan cost was serialized small-file I/O, not TMDb
|
||||
|
||||
Cold scan of the subset root: **975 s for 163 items**.
|
||||
|
||||
| Time | Share | Where |
|
||||
|---|---|---|
|
||||
| 763 s | 78 % | `download_cast_profile`: 3494 cast images, strictly serialized; ≈0.22 s each ≈ 0.19 s network-mount write + 0.03 s HTTP |
|
||||
| ~100 s | 10 % | ffmpeg probes (469 files, incl. 83 HDR `showinfo` passes) |
|
||||
| 73 s | 7 % | TMDb API layer: 217 uncached requests (27 s HTTP) **plus ~41 s writing per-request cache JSON files** to the network mount |
|
||||
| ~75 s | 8 % | covers / backdrops / season posters (same small-write cost) |
|
||||
| 0.3 s | — | filesystem discovery walk |
|
||||
|
||||
Notes:
|
||||
|
||||
- The TMDb disk cache itself is fine: once warm it serves 282 requests in
|
||||
0.8 s. Cache *reads* need no optimization.
|
||||
- The full cast of every title was downloaded sequentially, one tiny file per
|
||||
person (the full library has ~35k people records). Re-runs are cheap
|
||||
(exists-check), so this was a cold-scan-only cost — but it made the first
|
||||
scan of a new root take ~8× longer than everything else combined.
|
||||
|
||||
### F6 — Measured as noise (not worth effort)
|
||||
|
||||
- The 30-second rescan loop's tree walk: 3.5–6 s per pass over the full
|
||||
library (~1900 directories). Continuous but light.
|
||||
- `get_directory_size` per torrent: 0.1 s total in the scan.
|
||||
- TMDb disk-cache reads: sub-second per scan.
|
||||
- `trigger_scan` was dead code — nothing called it.
|
||||
- `_seen_mtimes` was updated *before* processing; a cancelled/failed scan
|
||||
permanently lost that update until the next restart.
|
||||
- The in-process probe cache was keyed by path only; a replaced file kept
|
||||
stale probe data until restart.
|
||||
|
||||
---
|
||||
|
||||
## 2. Implemented fixes
|
||||
|
||||
All proposals P1–P5 from the review were implemented, keeping the existing
|
||||
on-disk format unchanged (no blob storage — see Appendix A).
|
||||
|
||||
### F1 → merge-semantics upserts + deletion sync
|
||||
|
||||
- The `Upsert` event now carries `scanned: list[str]` — the media-root-relative
|
||||
torrent paths whose content was (re)scanned to build the item
|
||||
(`mediahive/models/events.py`; `_process_movies`/`_process_series` yield it).
|
||||
- `IndexStore.upsert_movie/upsert_series` (`mediahive/index_store.py`) merge a
|
||||
partial rebuild into the existing entry instead of replacing it:
|
||||
file entries belonging to scanned torrents are replaced, everything else is
|
||||
preserved, episodes/seasons emptied by the merge are dropped, same-episode
|
||||
multi-release files are unioned, and non-None scalar fields from the fresh
|
||||
scan win. The store broadcasts only when the merged result actually changed.
|
||||
- The TMDb dedupe collapse now folds duplicates through the same merge (the
|
||||
complete entry's scalars win), so a partial candidate can no longer delete a
|
||||
complete entry; the reverse TMDb-id map is fixed up incrementally instead of
|
||||
rebuilding both maps per upsert (also F4/P5).
|
||||
- **Deletion sync**: discovery collects the full set of candidate torrent
|
||||
paths; after a fully completed scan the scanner emits a new `Sync` event and
|
||||
`IndexStore.sync_torrent_paths` drops file entries whose torrent path is
|
||||
gone, cascading to empty episodes/seasons/items with proper removals.
|
||||
|
||||
### F2 → persisted scan state and probe cache
|
||||
|
||||
Three small JSON files under `.mediahive/` per root, loaded at scanner start
|
||||
and written atomically (tmp + rename) **only when changed**:
|
||||
|
||||
- `scan-state.json`: relpath → mtime. A restart over an unchanged library now
|
||||
discovers "0 new items" and finishes in walk time. Mtimes are committed
|
||||
**after** the scan completes successfully (fixes the pre-commit nit from F6):
|
||||
a cancelled/failed scan retries its items.
|
||||
- `probe-cache.json`: path → {mtime, size, probe fields} for every probed
|
||||
file, **failures included**. Keying by mtime+size makes it self-invalidating
|
||||
when a file is replaced (also fixes the stale-probe nit from F6). Non-plain
|
||||
paths (bluray:/concat: URIs) fail `stat` and stay memory-cached only.
|
||||
- `reel-state.json`: see F3 below.
|
||||
|
||||
### F3 → reel-state persistence, backoff, and queue gating
|
||||
|
||||
- `reel-state.json` records, per media folder (movies) or per episode
|
||||
(`folder#SxxEyy`), the video's mtime+size, status (`done`/`failed`),
|
||||
attempt count, and last-attempt timestamp.
|
||||
- `_reel_needed` gates both scan-time queueing and the worker:
|
||||
`done` entries are skipped; `failed` entries back off exponentially
|
||||
(6 h → 12 h → … capped at 1 week); entries with no record fall back to a
|
||||
cheap on-disk existence check, and existing reels are silently recorded as
|
||||
`done` so future scans take the cheap path. This ends both the infinite
|
||||
retries of unreadable files and the re-queueing of short videos.
|
||||
- The worker no longer rebroadcasts whole (stale) items: it sends narrow
|
||||
`MovieShowreel` / `EpisodeReel` events, and the store updates only the reel
|
||||
fields of the current entry (`set_movie_showreel` / `set_episode_reel`).
|
||||
- Reel state is persisted at scan finalize, on scanner `stop()`, and as soon
|
||||
as the reel queue drains (a crash between scans no longer loses records).
|
||||
|
||||
### F5 → bounded parallelism for downloads and TMDb fetches
|
||||
|
||||
- Cast-profile downloads: `asyncio.gather` with a semaphore of 8.
|
||||
- TMDb title lookups (movies and series) are prefetched in parallel
|
||||
(semaphore of 4) before the grouping loops, which then read the per-call
|
||||
caches.
|
||||
- Season-detail fetches are prefetched in parallel (semaphore of 4) before the
|
||||
season loop.
|
||||
- Per-item cover/backdrop/poster logic is unchanged (exists-check-fast when
|
||||
warm).
|
||||
|
||||
### Housekeeping (P5)
|
||||
|
||||
- Dead `trigger_scan` removed (`is_scanning` kept).
|
||||
- TMDb-id index bookkeeping is incremental (see F1 above).
|
||||
- `_rebuild_tmdb_indexes` remains only for snapshot load and dedupe.
|
||||
|
||||
---
|
||||
|
||||
## 3. Measured results
|
||||
|
||||
Subset root (163 items, 121 movies / 40 series entries, 469 video files),
|
||||
same network mount:
|
||||
|
||||
| Scenario | Before | After |
|
||||
|---|---|---|
|
||||
| Touch one season of a 5-season series | other 4 seasons vanish until next full scan | 0.3 s rescan, one partial upsert (`scanned=[S04]`), store keeps all 5 seasons |
|
||||
| Steady rescan, nothing changed (same process) | 0.2 s | 0.2 s, 0 upserts, no writes |
|
||||
| Warm restart, unchanged library (fresh process) | 56.5 s (94 % ffmpeg re-probes) + 467 reel tasks queued | **0.2 s, 0 upserts, reel queue 0** |
|
||||
| First scan with warm TMDb cache but no probe cache | 56.5 s | 51.6 s once — writes `probe-cache.json` (469 records, 207 KB); subsequent runs skip all probing |
|
||||
| Permanently unreadable files (3 DoVi/libplacebo movies) | retried on every scan and every startup, error broadcast each time | recorded as failed once, skipped within backoff |
|
||||
| Cold scan, nothing cached | 975 s | not re-measured end-to-end; the dominant terms are now 8-way parallel (cast images: 3494 downloads measured at 3.7 s when warm) |
|
||||
| Deleted torrent | stayed in index forever | removed on the next completed scan via `Sync` |
|
||||
|
||||
Unit-level checks (synthetic `IndexStore` + scanner state, no filesystem
|
||||
library involved): partial upsert preserves untouched seasons and replaces
|
||||
rescanned torrent files; multi-release episode union; sync removal cascades;
|
||||
single-episode reel updates; probe-cache save/load roundtrip; reel backoff
|
||||
math; reel-state persistence roundtrip across scanner instances.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A — Blob storage options: scan-time vs runtime concerns
|
||||
|
||||
**Status: reference only — not implemented.** The current on-disk format
|
||||
(one file per artifact) was deliberately kept. This appendix stays as
|
||||
documentation of the options if write amplification or file counts ever
|
||||
become an operational issue.
|
||||
|
||||
The two concerns have opposite constraints, so they should be decided
|
||||
separately:
|
||||
|
||||
- **Scan-time data** is written and read only by the scanner. Nothing in the
|
||||
server serves it. Storage format is therefore a pure implementation detail
|
||||
and can be changed freely.
|
||||
- **Runtime data** is delivered to the frontend as plain files via
|
||||
`/api/assets/{root}/{movies|series|people}/{path}` (covers, posters,
|
||||
backdrops, person photos) and `/api/media/{root}/{path}` (media files), with
|
||||
etag/range streaming. Anything that replaces files here must keep an HTTP
|
||||
serving story working.
|
||||
|
||||
### A.1 Artifact inventory (measured)
|
||||
|
||||
| Artifact | Class | Avg size | Count (full library) | Total | Written |
|
||||
|---|---|---|---|---|---|
|
||||
| TMDb response cache JSON | scan-time | 25 KB (median 9 KB) | ~8,500 | ~215 MB | on every uncached API request |
|
||||
| Probe results / scan state / reel state | scan-time | ~0.5 KB/record | ~3,000 records | ~1.5 MB | per processed file |
|
||||
| Person photos | runtime | 11.4 KB | ~22,500 | ~262 MB | once per person |
|
||||
| `cover.jpg` / season posters | runtime | ~77 KB | ~2,100 | ~164 MB | once per title/season |
|
||||
| `backdrop.jpg` | runtime | ~147 KB | ~1,460 | ~220 MB | once per title |
|
||||
| Reel clips (WebM/AV1) | runtime | ~450 KB | ~8,700 | ~4.0 GB | once per title/episode |
|
||||
| `index.json` snapshot | both | 7.5 MB | 1 | 7.5 MB | debounced, only when dirty |
|
||||
|
||||
Reference point for the write-amplification math: on the network mount one
|
||||
small-file write costs ~0.2 s, while a single 7.5 MB sequential write costs
|
||||
the same ~0.2 s (~40 MB/s). So ~31,000 tiny files ≈ 1.7 hours of serialized
|
||||
write time, versus ~6 s for the same bytes as one bulk dump.
|
||||
|
||||
### A.2 Scan-time blob store (TMDb cache, probe cache, scan state)
|
||||
|
||||
Nothing here is served, so the only requirement is fast lookup + cheap
|
||||
persistence. Two workable shapes:
|
||||
|
||||
- **RAM map + debounced atomic dump.** Plain dicts keyed by request hash /
|
||||
file path, dumped as one binary file (length-prefixed msgspec or JSON blob,
|
||||
optionally zstd-compressed) with tmp-write + rename, on the same
|
||||
dirty-flag + debounce discipline `index.json` already uses. Effects: the
|
||||
~8,500 individual cache writes collapse into a handful of bulk flushes;
|
||||
warm lookups become dict hits with zero filesystem calls. TMDb JSON
|
||||
compresses ~10× (215 MB → ~20–25 MB), so a full dump is a sub-second write.
|
||||
Caveat: holding all responses parsed in RAM costs ~200 MB for the full
|
||||
library; storing raw response *bytes* and parsing lazily, or capping to
|
||||
entries referenced by known index items, keeps this modest.
|
||||
- **SQLite (stdlib, WAL mode).** One database file, incremental commits, crash
|
||||
safety without full dumps, and kernel page cache instead of explicit RAM
|
||||
management. Better fit if the cache is allowed to grow unbounded, at the
|
||||
price of slightly more code.
|
||||
|
||||
Either way, keep the existing cache *semantics* unchanged: cache HTTP-level
|
||||
failures, never cache network errors. With persisted scan state in place the
|
||||
TMDb cache becomes write-rarely (new items only), which further lowers the
|
||||
value of elaborate engineering here — the simple dump is likely enough.
|
||||
|
||||
### A.3 Runtime-served artifacts
|
||||
|
||||
- **Reels, covers, backdrops, season posters: keep as files.** They are few
|
||||
per title, tens-to-hundreds of KB, written exactly once, and benefit from
|
||||
the existing etag/range file serving. No write-amplification problem.
|
||||
- **Person photos** (~22.5k files × 11.4 KB) are the one runtime class where
|
||||
tiny files hurt at scan time. Three options, in increasing invasiveness:
|
||||
1. **Keep files, fix only the scan-time behavior** — bounded-parallel
|
||||
downloads, optionally capped to top-N billed cast. Zero changes to
|
||||
serving; the cold-scan cost drops ~8× but the file count stays.
|
||||
*(This is the option currently implemented.)*
|
||||
2. **Blob db + serve from the db.** Person photos move into the same store
|
||||
as above; the assets handler gains one branch for the `people` asset
|
||||
type that streams bytes from the db instead of the filesystem.
|
||||
Eliminates all 22.5k tiny files. If full RAM residency (262 MB) is
|
||||
undesirable, use SQLite and let the page cache handle it.
|
||||
3. **Hybrid lazy materialization.** The db is authoritative at scan time
|
||||
(no tiny writes during scans); the assets handler writes the photo to the
|
||||
conventional path on first request and serves it as a file thereafter.
|
||||
Serving logic and URLs stay unchanged; disk usage appears only for
|
||||
people actually viewed.
|
||||
|
||||
Note that `index.json` itself is already the right shape: one atomic 7.5 MB
|
||||
file, rewritten only when dirty. The goal for everything else is simply to
|
||||
reach the same shape per concern.
|
||||
|
||||
### A.4 Recommendation (if revisited)
|
||||
|
||||
| Concern | Recommended treatment |
|
||||
|---|---|
|
||||
| TMDb response cache | RAM map + debounced single-file dump (SQLite if growth matters) |
|
||||
| Probe cache, seen-mtimes, reel-failure records | same dump mechanism, separate small files *(currently: three small JSON files, written only when dirty)* |
|
||||
| Person photos | option 1 now (parallel + lazy fetching); option 2 or 3 if file count becomes an operational issue |
|
||||
| Covers, posters, backdrops, reels | unchanged — plain files |
|
||||
| `index.json` | unchanged |
|
||||
Reference in New Issue
Block a user