mediahive/docs/scanning-review.md

Scanner review — findings, fixes, and measured results

Date: 2026-09-03 (review); fixes implemented same day. Scope: mediahive/hivescan/*, mediahive/index_store.py, mediahive/root_registry.py, mediahive/models/events.py Method: code review plus instrumented runs of the real RootScanner (monkeypatched timers around every ffmpeg invocation, HTTP request, and filesystem primitive).

Structure of this document:

• Section 1 records the findings as measured against the pre-fix code (line references are from that revision).
• Section 2 describes the fixes that were implemented for each finding.
• Section 3 gives before/after measurements.
• Appendix A (blob storage options) is kept for reference only; it was explicitly decided not to change the on-disk storage format for now.

Test environment (details omitted intentionally): the media library lives on a network-mounted filesystem (SMB/CIFS). Two roots were measured:

• Subset root: 163 torrents (121 movies, 40 series entries), stale/empty index.
• Full library root: existing index with 1127 movies, 159 series, 1608 episodes, ~2983 video files, ~35k people records.

Environment characteristic that dominates several measurements: stat/scandir/exists on the network mount are ~0.1 ms (attribute caching works), but every small file write costs ~0.2 s (synchronous write-through). The scanner writes thousands of small files into .mediahive/ on that mount.

---

1. Findings (pre-fix)

F1 — Partial rescan replaces whole entries (the "disappearing seasons" bug)

This is a correctness bug, not a performance issue.

Chain of events:

1. Discovery mtime-gates per torrent directory/file. Touching one season directory of a series yields a downloads list containing only that season.
2. _process_series rebuilds the Series object from only the items it was given.
3. IndexStore.upsert_series replaced the entire entry and broadcast the partial series to all connected clients.

Measured end-to-end on a 5-season series (5 separate season torrents, 110 episodes):

full scan        -> index entry seasons [1,2,3,4,5] (110 episodes)
touch season 4   -> rescan 0.2 s, emits ONE upsert: seasons [4] (22 episodes)
                 -> index entry is now seasons [4] — seasons 1-3,5 gone

The missing seasons returned only when a scan happened to include all seasons again — in practice the next process restart, because the seen-mtimes map (F2) was memory-only and forced a full rediscovery at startup.

Same bug class, other variants:

• Movies: a touched version directory dropped the other versions of the same movie from the listing.
• TMDb dedupe collapse: when a partial entry arrived under a different item id for an already-known TMDb id, the old complete entry was explicitly deleted.
• Deletions were never detected: discovery only ever added to _seen_mtimes; nothing emitted removals. A torrent deleted from disk stayed in the index forever.

F2 — No persistent scan state: every restart was a full reprocess

All scanner state was process memory: _seen_mtimes, the ffmpeg probe cache, and the episode/playable-file/bluray-probe caches.

Consequences measured:

• Steady-state rescan within one process: 0.2 s (subset root, nothing changed) — mtime gating worked fine while the process lived.
• Warm-restart scan (fresh process caches, all disk caches warm): 56.5 s for the same 163 items, of which 52.9 s (94 %) was re-running ffmpeg probes on all 469 video files (21 s ffmpeg -i + 32 s showinfo passes on HDR files). TMDb was 100 % disk-cache hits and cost 0.8 s total.
• Scaled to the full library: every application restart re-probed ~2983 files ≈ 6–8 minutes of sequential ffmpeg, during which the whole index was re-derived and re-upserted item by item (see F4).

F3 — Preview (showreel) generation: restart storms and infinite retries

• Every scan that included an item enqueued all of its reel tasks, whether or not the reels existed. Existence was only checked later by the serial worker. A full scan of the subset root enqueued 467 tasks; the full library would enqueue ~2700.
• Nothing was persisted about the queue. A restart before the queue drained started everything over.
• Failures were never recorded. In the drain test, 3 of 18 movies failed deterministically (DoVi profile 7 titles require libplacebo tonemapping, which fails on GPU-less machines; one file has a matroska demux error). The same files were retried on every subsequent drain — ~1.5 s of probing plus crop detection plus an error task broadcast to every client, forever.
• The reel-existence check required all five reels; short videos legitimately produce fewer, so they were treated as "missing" and re-queued on every scan, generating nothing new each time.
• Measured generation pace with software AV1 encoding: ~16–28 s per movie (5 clips), ~4–5 s per episode clip.
• The reel worker rebroadcast whole items built from stale scan data, clobbering newer store state.

F4 — Degraded operation while a full scan is in progress

• Items were re-upserted one by one as processed; combined with F1, any partial rescan interleaved with normal use made listings lose data until the next restart.
• The showreel worker broadcast an error task for every permanent failure on every scan (F3), producing user-visible noise.
• The index snapshot was rewritten every 5 s while dirty; with 1286+ items that is a ~7.5 MB serialize + write per flush, continuously, for the duration of a scan.
• upsert_* rebuilt the TMDb-id lookup maps on every upsert — O(n²) per scan. Measured negligible; fixed anyway as part of the merge work.

F5 — Cold-scan cost was serialized small-file I/O, not TMDb

Cold scan of the subset root: 975 s for 163 items.

Time	Share	Where
763 s	78 %	download_cast_profile: 3494 cast images, strictly serialized; ≈0.22 s each ≈ 0.19 s network-mount write + 0.03 s HTTP
~100 s	10 %	ffmpeg probes (469 files, incl. 83 HDR showinfo passes)
73 s	7 %	TMDb API layer: 217 uncached requests (27 s HTTP) plus ~41 s writing per-request cache JSON files to the network mount
~75 s	8 %	covers / backdrops / season posters (same small-write cost)
0.3 s	—	filesystem discovery walk

Notes:

• The TMDb disk cache itself is fine: once warm it serves 282 requests in 0.8 s. Cache reads need no optimization.
• The full cast of every title was downloaded sequentially, one tiny file per person (the full library has ~35k people records). Re-runs are cheap (exists-check), so this was a cold-scan-only cost — but it made the first scan of a new root take ~8× longer than everything else combined.

F6 — Measured as noise (not worth effort)

• The 30-second rescan loop's tree walk: 3.5–6 s per pass over the full library (~1900 directories). Continuous but light.
• get_directory_size per torrent: 0.1 s total in the scan.
• TMDb disk-cache reads: sub-second per scan.
• trigger_scan was dead code — nothing called it.
• _seen_mtimes was updated before processing; a cancelled/failed scan permanently lost that update until the next restart.
• The in-process probe cache was keyed by path only; a replaced file kept stale probe data until restart.

---

2. Implemented fixes

All proposals P1–P5 from the review were implemented, keeping the existing on-disk format unchanged (no blob storage — see Appendix A).

F1 → merge-semantics upserts + deletion sync

• The Upsert event now carries scanned: list[str] — the media-root-relative torrent paths whose content was (re)scanned to build the item (mediahive/models/events.py; _process_movies/_process_series yield it).
• IndexStore.upsert_movie/upsert_series (mediahive/index_store.py) merge a partial rebuild into the existing entry instead of replacing it: file entries belonging to scanned torrents are replaced, everything else is preserved, episodes/seasons emptied by the merge are dropped, same-episode multi-release files are unioned, and non-None scalar fields from the fresh scan win. The store broadcasts only when the merged result actually changed.
• The TMDb dedupe collapse now folds duplicates through the same merge (the complete entry's scalars win), so a partial candidate can no longer delete a complete entry; the reverse TMDb-id map is fixed up incrementally instead of rebuilding both maps per upsert (also F4/P5).
• Deletion sync: discovery collects the full set of candidate torrent paths; after a fully completed scan the scanner emits a new Sync event and IndexStore.sync_torrent_paths drops file entries whose torrent path is gone, cascading to empty episodes/seasons/items with proper removals.

F2 → persisted scan state and probe cache

Three small JSON files under .mediahive/ per root, loaded at scanner start and written atomically (tmp + rename) only when changed:

• scan-state.json: relpath → mtime. A restart over an unchanged library now discovers "0 new items" and finishes in walk time. Mtimes are committed after the scan completes successfully (fixes the pre-commit nit from F6): a cancelled/failed scan retries its items.
• probe-cache.json: path → {mtime, size, probe fields} for every probed file, failures included. Keying by mtime+size makes it self-invalidating when a file is replaced (also fixes the stale-probe nit from F6). Non-plain paths (bluray:/concat: URIs) fail stat and stay memory-cached only.
• reel-state.json: see F3 below.

F3 → reel-state persistence, backoff, and queue gating

• reel-state.json records, per media folder (movies) or per episode (folder#SxxEyy), the video's mtime+size, status (done/failed), attempt count, and last-attempt timestamp.
• _reel_needed gates both scan-time queueing and the worker: done entries are skipped; failed entries back off exponentially (6 h → 12 h → … capped at 1 week); entries with no record fall back to a cheap on-disk existence check, and existing reels are silently recorded as done so future scans take the cheap path. This ends both the infinite retries of unreadable files and the re-queueing of short videos.
• The worker no longer rebroadcasts whole (stale) items: it sends narrow MovieShowreel / EpisodeReel events, and the store updates only the reel fields of the current entry (set_movie_showreel / set_episode_reel).
• Reel state is persisted at scan finalize, on scanner stop(), and as soon as the reel queue drains (a crash between scans no longer loses records).

F5 → bounded parallelism for downloads and TMDb fetches

• Cast-profile downloads: asyncio.gather with a semaphore of 8.
• TMDb title lookups (movies and series) are prefetched in parallel (semaphore of 4) before the grouping loops, which then read the per-call caches.
• Season-detail fetches are prefetched in parallel (semaphore of 4) before the season loop.
• Per-item cover/backdrop/poster logic is unchanged (exists-check-fast when warm).

Housekeeping (P5)

• Dead trigger_scan removed (is_scanning kept).
• TMDb-id index bookkeeping is incremental (see F1 above).
• _rebuild_tmdb_indexes remains only for snapshot load and dedupe.

---

3. Measured results

Subset root (163 items, 121 movies / 40 series entries, 469 video files), same network mount:

Scenario	Before	After
Touch one season of a 5-season series	other 4 seasons vanish until next full scan	0.3 s rescan, one partial upsert (scanned=[S04]), store keeps all 5 seasons
Steady rescan, nothing changed (same process)	0.2 s	0.2 s, 0 upserts, no writes
Warm restart, unchanged library (fresh process)	56.5 s (94 % ffmpeg re-probes) + 467 reel tasks queued	0.2 s, 0 upserts, reel queue 0
First scan with warm TMDb cache but no probe cache	56.5 s	51.6 s once — writes probe-cache.json (469 records, 207 KB); subsequent runs skip all probing
Permanently unreadable files (3 DoVi/libplacebo movies)	retried on every scan and every startup, error broadcast each time	recorded as failed once, skipped within backoff
Cold scan, nothing cached	975 s	not re-measured end-to-end; the dominant terms are now 8-way parallel (cast images: 3494 downloads measured at 3.7 s when warm)
Deleted torrent	stayed in index forever	removed on the next completed scan via Sync

Unit-level checks (synthetic IndexStore + scanner state, no filesystem library involved): partial upsert preserves untouched seasons and replaces rescanned torrent files; multi-release episode union; sync removal cascades; single-episode reel updates; probe-cache save/load roundtrip; reel backoff math; reel-state persistence roundtrip across scanner instances.

---

Appendix A — Blob storage options: scan-time vs runtime concerns

Status: reference only — not implemented. The current on-disk format (one file per artifact) was deliberately kept. This appendix stays as documentation of the options if write amplification or file counts ever become an operational issue.

The two concerns have opposite constraints, so they should be decided separately:

• Scan-time data is written and read only by the scanner. Nothing in the server serves it. Storage format is therefore a pure implementation detail and can be changed freely.
• Runtime data is delivered to the frontend as plain files via /api/assets/{root}/{movies|series|people}/{path} (covers, posters, backdrops, person photos) and /api/media/{root}/{path} (media files), with etag/range streaming. Anything that replaces files here must keep an HTTP serving story working.

A.1 Artifact inventory (measured)

Artifact	Class	Avg size	Count (full library)	Total	Written
TMDb response cache JSON	scan-time	25 KB (median 9 KB)	~8,500	~215 MB	on every uncached API request
Probe results / scan state / reel state	scan-time	~0.5 KB/record	~3,000 records	~1.5 MB	per processed file
Person photos	runtime	11.4 KB	~22,500	~262 MB	once per person
cover.jpg / season posters	runtime	~77 KB	~2,100	~164 MB	once per title/season
backdrop.jpg	runtime	~147 KB	~1,460	~220 MB	once per title
Reel clips (WebM/AV1)	runtime	~450 KB	~8,700	~4.0 GB	once per title/episode
index.json snapshot	both	7.5 MB	1	7.5 MB	debounced, only when dirty

Reference point for the write-amplification math: on the network mount one small-file write costs ~0.2 s, while a single 7.5 MB sequential write costs the same ~0.2 s (~40 MB/s). So ~31,000 tiny files ≈ 1.7 hours of serialized write time, versus ~6 s for the same bytes as one bulk dump.

A.2 Scan-time blob store (TMDb cache, probe cache, scan state)

Nothing here is served, so the only requirement is fast lookup + cheap persistence. Two workable shapes:

• RAM map + debounced atomic dump. Plain dicts keyed by request hash / file path, dumped as one binary file (length-prefixed msgspec or JSON blob, optionally zstd-compressed) with tmp-write + rename, on the same dirty-flag + debounce discipline index.json already uses. Effects: the ~8,500 individual cache writes collapse into a handful of bulk flushes; warm lookups become dict hits with zero filesystem calls. TMDb JSON compresses ~10× (215 MB → ~20–25 MB), so a full dump is a sub-second write. Caveat: holding all responses parsed in RAM costs ~200 MB for the full library; storing raw response bytes and parsing lazily, or capping to entries referenced by known index items, keeps this modest.
• SQLite (stdlib, WAL mode). One database file, incremental commits, crash safety without full dumps, and kernel page cache instead of explicit RAM management. Better fit if the cache is allowed to grow unbounded, at the price of slightly more code.

Either way, keep the existing cache semantics unchanged: cache HTTP-level failures, never cache network errors. With persisted scan state in place the TMDb cache becomes write-rarely (new items only), which further lowers the value of elaborate engineering here — the simple dump is likely enough.

A.3 Runtime-served artifacts

• Reels, covers, backdrops, season posters: keep as files. They are few per title, tens-to-hundreds of KB, written exactly once, and benefit from the existing etag/range file serving. No write-amplification problem.
• Person photos (~22.5k files × 11.4 KB) are the one runtime class where tiny files hurt at scan time. Three options, in increasing invasiveness:
  1. Keep files, fix only the scan-time behavior — bounded-parallel downloads, optionally capped to top-N billed cast. Zero changes to serving; the cold-scan cost drops ~8× but the file count stays. (This is the option currently implemented.)
  2. Blob db + serve from the db. Person photos move into the same store as above; the assets handler gains one branch for the people asset type that streams bytes from the db instead of the filesystem. Eliminates all 22.5k tiny files. If full RAM residency (262 MB) is undesirable, use SQLite and let the page cache handle it.
  3. Hybrid lazy materialization. The db is authoritative at scan time (no tiny writes during scans); the assets handler writes the photo to the conventional path on first request and serves it as a file thereafter. Serving logic and URLs stay unchanged; disk usage appears only for people actually viewed.

Note that index.json itself is already the right shape: one atomic 7.5 MB file, rewritten only when dirty. The goal for everything else is simply to reach the same shape per concern.

A.4 Recommendation (if revisited)

Concern	Recommended treatment
TMDb response cache	RAM map + debounced single-file dump (SQLite if growth matters)
Probe cache, seen-mtimes, reel-failure records	same dump mechanism, separate small files (currently: three small JSON files, written only when dirty)
Person photos	option 1 now (parallel + lazy fetching); option 2 or 3 if file count becomes an operational issue
Covers, posters, backdrops, reels	unchanged — plain files
index.json	unchanged