Implement database rotation with n days retention #2

2026-09-02T16:45:57Z

LeoVasanko commented

2026-09-02 16:45:57 +00:00

Automatically erase history, keep as separate timestamped files.
Full history may be recovered by concatenation of the files.
Only applied at database opening time.
Enable by Kanta(retention=...), use with care (experimental feature).

- Automatically erase history, keep as separate timestamped files. - Full history may be recovered by concatenation of the files. - Only applied at database opening time. - Enable by Kanta(retention=...), use with care (experimental feature).

LeoVasanko added 8 commits 2026-09-02 16:45:57 +00:00

Add homepage bcf8ccb830

Replace jsondiff producer with own diff; support list edits in replay 379b0de1dc

compute_diff now emits a simple subset of jsondiff's marshaled syntax:
plain key assignment for adds and full-value changes (lists replaced
wholesale), $delete for removed object keys, and $replace only when a
dict replaces a non-dict. Removed keys are always $delete, even when the
object becomes empty (jsondiff emitted $replace: {}).

apply_diff (single implementation in serialization.base, used by both
replay and patch_state) additionally accepts jsondiff-produced patches:
positional $insert/$delete and per-index nested diffs on lists, plus
$ escaping of user keys and values per the marshal format.

jsondiff moves to the dev dependency group; it is used only by tests
that verify patch compatibility in both directions.

Simplify diff producer: no $replace, no value escaping 1bf52629a3

The producer now emits only plain key assignment (full-value replacement
for scalars, lists and type changes, including dict-over-non-dict) and
$delete for removed keys. $-prefixed keys are still escaped as $$;
values are stored verbatim.

The consumer remains compatible with jsondiff-produced diffs ($replace,
positional $insert/$delete, per-index list diffs). A dict diff against
a list state is treated as jsondiff list-ops only when every key is a
command or integer position, otherwise as our wholesale replacement.
Unescaping is narrowed to the $$ prefix so verbatim single-$ values
pass through untouched.

docs: add database rotation design 1e43f29eec

filelock: add replace_content for in-place locked rewrite 010b690b47

Rotation rewrites the database file while holding the exclusive lock;
flock follows the open file description across ftruncate, and on Windows
in-place rewrite avoids share-mode rename restrictions. Also update
rotation doc timestamp format to ISO basic with microseconds.

rotation: implement retention-based history rotation 5a5f8b011c

New kanta.rotation module plans rotation from in-memory bytes: walks
snapshots backwards to a replay base predating the cutoff, validates
replayed state against every snapshot in range, and rebuilds the file as
leading cutoff snapshot + re-framed retained changes (+ final snapshot
when enough changes survived). KantaImpl.open executes the plan under
the exclusive lock before replay: copy2 aside to {stem}@{ts}.kantadb,
in-place locked rewrite with fdatasync, then trim the rotated copy to
the dropped-history prefix. Enabled via Kanta(retention=timedelta).

rotation: tests, replay-base cutoff state fix, formatting b8285e9170

- tests/test_rotation.py covers both framers: history split, idempotent
  reopen, no-op cases, aged-out reduction to a single snapshot, internal
  snapshot validation, extension-agnostic rotated naming, and the
  disabled default.
- Fix plan_rotation to seed the cutoff state from the replay base
  snapshot (previously asserted when the base already covered the cutoff).
- docs: snapshots carry the schema version in effect at their position,
  keeping replay/migration behavior identical to the unrotated file.

rotation: second-precision rotated filenames, int-days retention 3074de2950

- Rotated filename timestamp is now ISO basic at second precision
  (20260902T143000Z); the exact cutoff ts remains inside the files.
- Kanta(retention=N) accepts a plain int as number of days.

LeoVasanko merged commit aee6c13996 into main

2026-09-02 16:46:19 +00:00

LeoVasanko deleted branch rotation

2026-09-02 16:46:19 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LeoVasanko/kanta#2