35 Commits
Author SHA1 Message Date
LeoVasanko ec8695ba0c Filter snapshot lines through --grep like change records
Snapshot indicator lines printed unconditionally under --grep, suggesting
the snapshot matched. Now the snapshot's full state is flattened into the
same (path, value) entries change records are matched against, and the
snapshot line is suppressed unless every pattern matches. Snapshots
explicitly selected with -r s<N> still print unconditionally.
2026-09-21 19:13:19 +00:00
LeoVasanko a4efbca55b Clarify diagnostic log messages; simplify describe_callback fallback
Unnamed callables (partials, callable instances) are described by type
name only: docstrings are shown only for named callables, avoiding
misleading class docstrings in failure messages.

Diagnostic messages revised for clarity when mixed with application
logs; logger.exception() messages no longer repeat the exception text,
which the traceback already shows.
2026-09-16 02:55:10 +00:00
LeoVasanko 44c1ed191e Configure kanta event loggers at import time, inheriting the root level.
configure_logging() now runs with default arguments when kanta.logging
is imported, attaching a plain stderr handler with propagate=False to
the event loggers (kanta.bootstrap/migration/transaction) that carry
Kanta-rendered output.  No levels are set by default, so event output
inherits the effective root level: a framework switching root between
INFO in development and WARNING in production governs Kanta output too.

Other configure_logging changes: channel enable flags use
logger.disabled (propagate toggling no longer silences now that event
loggers have their own handler), skiproot=False removes Kanta's
handler and re-enables propagation so the root logger renders event
output, and debug=True lifts only the DEBUG-emitting loggers
(bootstrap, migration) to DEBUG instead of setting a level on the
"kanta" parent.
2026-09-16 02:16:58 +00:00
LeoVasanko a41f34d332 Route all diagnostic logging through the plain "kanta" logger.
Module loggers used __name__, splitting diagnostics across eight
module-named loggers and colliding kanta.transaction with the
transaction event channel.  Diagnostics (integrity errors, flush
failures, rotation notes) are few; they now all go through the
"kanta" logger, following the application's root logging
configuration like any ordinary library output.
2026-09-16 02:16:58 +00:00
LeoVasanko 7a7716ec7b Make kanta.diff/patch functions public API. 2026-09-13 00:29:02 +00:00
LeoVasanko c608e749a7 Implement kanta --grep (intelligent search string) with highlight marks. 2026-09-13 00:21:26 +00:00
LeoVasanko 8e780d7ee4 Show package version number on kanta CLI 2026-09-12 22:15:17 +00:00
LeoVasanko 2e6f48bac5 Implement support for NO_COLOR/FORCE_COLOR env with isatty and journald checks for autodetection. 2026-09-12 22:07:37 +00:00
LeoVasanko ab5b7584c9 Updated docs 2026-09-02 17:34:20 +00:00
LeoVasanko d33f3f9c2f Add @kanta.validate integrity-validation callbacks
Validators receive the live data object (and optionally the Kanta
instance) and raise on inconsistency. They run after replay/migrations
during open() and after each transaction before the change is queued;
a failure rolls back the transaction or aborts the open. Multiple
validators run in registration order until the first failure. Sync-only,
since transactions are synchronous.
2026-09-02 17:02:14 +00:00
LeoVasanko aee6c13996 Implement database rotation with n days retention (#2)
- Automatically erase history, keep as separate timestamped files.
- Full history may be recovered by concatenation of the files.
- Only applied at database opening time.
- Enable by Kanta(retention=...), use with care (experimental feature).
2026-09-02 16:46:19 +00:00
LeoVasanko 94c5ddeaba Inject state dicts by name (prev/state), rename MigrationResult to MigrationReport.
State dict injection now works by parameter name (prev/state, annotation
not checked) with DictPrev/DictState tags taking precedence; DictPre/DictPost
remain as aliases. MigrationResult is renamed to MigrationReport with fields
original/version/applied; the old type alias and a deprecated .migrations
property remain. Old symbols stay covered by the original tests; new-style
tests import from the kanta root. Includes some unrelated ruff formatting.
2026-08-27 14:50:34 +00:00
LeoVasanko 4ef74e027f Re-export from kanta configure_logging. 2026-08-27 13:41:31 +00:00
LeoVasanko 6839b48f6d Support filesystem paths in --data, --migrations, and --kanta arguments. 2026-08-13 21:31:21 +00:00
LeoVasanko 955fdd8e1c Locate only the current Python version's site-packages in nearby .venv dirs. 2026-08-13 21:27:14 +00:00
LeoVasanko 20694576c7 Scope sys.path additions around dynamic imports and include nearby .venv site-packages. 2026-08-13 21:23:28 +00:00
LeoVasanko 8be44bd490 Add CWD to sys.path so CLI can import target modules. 2026-08-13 21:12:59 +00:00
LeoVasanko f8a0a85158 Pretty-print snapshot lines and drop microseconds from CLI timestamps. 2026-08-13 21:12:33 +00:00
LeoVasanko 8e436295aa Add experimental kanta CLI for inspecting databases. 2026-08-13 18:44:46 +00:00
LeoVasanko 0b8c1d2da9 Ruff 2026-08-11 01:34:39 +00:00
LeoVasanko 33a8c07043 Make tests follow the earlier change of merging version and msgspec migration records into one. 2026-08-11 01:34:06 +00:00
LeoVasanko be3acaed3e Smarter change record formatting to 80ch wide, shortened with ellipsis character rather than three dots. 2026-08-11 01:21:34 +00:00
LeoVasanko f489216c2a All remaining events through logformat, cleaner migration message, prettier demo. 2026-08-07 17:12:29 +00:00
LeoVasanko 08f3c44f1f Group all migration events into one row migrate:vN (if version changed) OR migrate:msgspec
- Log as a single event
- Logging config has debug parameter to lower level to DEBUG, showing migration diffs
2026-08-07 16:20:56 +00:00
LeoVasanko e0046ae9d9 Cleanup 2026-08-07 16:10:46 +00:00
LeoVasanko c101f187d8 Implement richer, fully customizable logging; customizable timestamps (#1)
- `@kanta.logemit` handler for completely customizable logging output, with `LogEvent` structure and `kanta.tty.Line` helper to create colorized text and fixed width fields
- `configure_logging(diff=False)` to disable diff display globally (supplementing per-transaction `logdiff=False`)
- `transaction(extra: Any = ...)` for passing extra strings or custom metadata to logs
- `@kanta.clock` to provide user controlled clock for deterministic database outputs
- Added a demo script that shows basic functions, migrations, logfmt etc.
2026-08-07 15:08:28 +00:00
LeoVasanko 3a56bfbb10 Add kanta.bootstrap logger and configurable logging setup
- Bootstrap records are now logged via kanta.bootstrap at INFO level.
- Existing databases log 'Using <path>' at DEBUG on kanta.bootstrap.
- New databases log 'Created <path>' at INFO on kanta.bootstrap.
- Renamed loggers: kanta.changes -> kanta.transaction, kanta.migrations -> kanta.migration.
- configure_logging() gains bootstrap/migration/transaction/skiproot kwargs.
- Default configure_logging() attaches a no-prefix stderr handler to kanta and stops propagation.
- With skiproot=False, child logger propagation flags are still applied but kanta itself is left untouched.
- Updated tests and docstrings.
2026-06-20 19:10:58 +00:00
LeoVasanko 55fa475a13 Log new database creation, bootstrap like a transaction. 2026-06-20 18:15:08 +00:00
LeoVasanko 753b7eba86 Add separate migration logging, logmigr callback, and log= override
- Split transaction logger (kanta.changes) and migration logger (kanta.migrations).
- Migrations.apply() now returns MigrationResult instead of logging.
- Kanta.open() emits one info summary per DB and debug transaction per migration.
- Add open(log=...) to suppress/redirect default migration logging.
- Add @kanta.logmigr callback for custom migration logging/summaries.
- Add transaction(log=...) to suppress/redirect transaction logging.
- Update tests for the new MigrationResult API and logging behaviour.
2026-06-15 23:13:03 +00:00
LeoVasanko 42789e6619 Revised migration context: don't store changerecord unless something was changed, and always snapshot if and after changes done. 2026-06-15 22:14:03 +00:00
LeoVasanko c4726e6728 Stricter versioning: always store migrate version record, file must be within versions included in migrations. Log only for migrations that made changes. Allow deleting older migration functions when no longer required. 2026-06-15 03:09:45 +00:00
LeoVasanko 66e92739ab Refactor Migrations internals, add caching to avoid reloading per each Kanta instance. Naming changed from MigrationRegistry to Migration, module from migrate to kanta.migrations. 2026-06-15 01:12:42 +00:00
LeoVasanko 4dc2f0648e Fix creation of new database, ensuring that a bootstrap record of the initial state is always written. 2026-06-15 00:27:49 +00:00
LeoVasanko bec4635460 Add kanta.ctx SimpleNamespace for user variables. This is passed to migration functions if they take a second argument. Remove the old MigrationCtx system. 2026-06-14 19:54:23 +00:00
LeoVasanko c04a245366 Add read only mode that doesn't use locking. Useful for inspection while the database is in use or when no changes are intended (except in RAM). 2026-06-14 19:35:06 +00:00
50 changed files with 7262 additions and 661 deletions
+9 -79
View File
@@ -51,84 +51,14 @@ asyncio.run(main())
3. Let Kanta flush queued changes to disk in the background. 3. Let Kanta flush queued changes to disk in the background.
4. Use snapshots and replay for fast startup and full history. 4. Use snapshots and replay for fast startup and full history.
## Bootstrap and Open Modes ## Documentation
Kanta supports open-time bootstrap callbacks for initializing a brand-new - [Usage patterns](https://git.zi.fi/LeoVasanko/kanta/src/branch/main/docs/usage.md) — opening, data ownership, and lifecycle patterns
database before `open()` returns. - [Bootstrap and open modes](https://git.zi.fi/LeoVasanko/kanta/src/branch/main/docs/bootstrap.md) — seeding new databases, strict and read-only opens
- [Validation](https://git.zi.fi/LeoVasanko/kanta/src/branch/main/docs/validation.md) — `@kanta.validate` integrity checks on open and transactions
- [Migrations](https://git.zi.fi/LeoVasanko/kanta/src/branch/main/docs/migrations.md) — versioned schema evolution with `migrate_vN`
- [Retention and rotation](https://git.zi.fi/LeoVasanko/kanta/src/branch/main/docs/rotation.md) — bounding history to a time window
- [Fatal error handlers](https://git.zi.fi/LeoVasanko/kanta/src/branch/main/docs/fatal-errors.md) — observing background write failures
- [On-disk format](https://git.zi.fi/LeoVasanko/kanta/src/branch/main/docs/database.md) — record layout and invariants
Register bootstrap handlers with a decorator: Kanta in JSON mode (default) stores newline-delimited records, so transaction history is viewable in any text editor; MsgPack mode uses binary records with length and checksum to guard against corruption.
```python
kanta = Kanta("data.kantadb", Data())
@kanta.bootstrap(action="seed", user="system")
def seed_defaults(data) -> None:
data.users["admin"] = User(name="Admin")
await kanta.open()
```
You can also use `@kanta.bootstrap` with no arguments and async handlers:
```python
@kanta.bootstrap
async def bootstrap_async(data) -> None:
data.counter = 1
```
When multiple bootstrap handlers are registered:
- they run in registration order,
- exactly one bootstrap change record is queued,
- bootstrap metadata (`action`, `user`, `mtime`) is taken from the last
registration.
If any bootstrap handler raises, Kanta closes and removes the database file,
then re-raises the error.
`open()` also supports strict open mode:
```python
await kanta.open(create=False)
```
With `create=False`, open fails if the database file does not exist or is
empty.
## Fatal Error Handlers
Fatal background write errors can be observed with a decorator:
```python
import os
import signal
@kanta.fatal_error
async def on_fatal(err):
os.kill(os.getpid(), signal.SIGTERM) # Die
```
Multiple fatal handlers are supported and run in registration order.
## Migrations
Adding or removing a field and other such simple operations are automatic, but when the time comes to really change your data model, implement a `migrate_v1` function that converts your old data to the new form. This works on plain built-in dict and other types, to avoid needing to preserve old versions of your structs.
Pass a module (or import path) containing `migrate_vN` functions:
```python
kanta = Kanta("data.kantadb", Data(), migrations="myapp.migrations")
await kanta.open()
```
Kanta tracks migration version metadata automatically, and fast forwards your database to current version by running all the migrations needed while opening the database.
## On-Disk Format
Kanta in JSON mode (default) stores newline-delimited records. Transaction history is viewable by any simple text editor, and rollbacks to prior state are done by simply removing final lines (one per transaction)
MsgPack mode uses binary records with length and checksum to avoid data corruption.
- Change line: JSON object with metadata + `diff`
- Snapshot line: `SNAPSHOT { ... full state ... }`
See `docs/database.md` for format details and invariants.
+1
View File
@@ -0,0 +1 @@
demo.kantadb
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env -S uv run
import asyncio
import sys
from datetime import UTC, datetime, timedelta
from pathlib import Path
import msgspec
from kanta import Kanta, configure_logging
filename = Path(__file__).with_name("demo.kantadb")
# For demonstration purposes, we use "original v0" and "modified v1" in this same script
# Normally your app would only have the latest supported data model
class Data(msgspec.Struct): # type: ignore - intentionally redefined later
users: dict[str, dict] = {}
counter: int = 0
kanta_v0 = Kanta(filename, Data())
@kanta_v0.bootstrap
def bootstrap(data: Data) -> None:
"""Create the initial admin user."""
data.users["userid001"] = {"name": "Alice", "role": "admin"}
# Redefinition to simulate new version
class Data(msgspec.Struct):
users: dict[str, dict] = {}
total: int = 0 # Replaces old counter field
lang: str = "en" # New field
def migrate_v1(d: dict) -> None:
"""Rename counter to total"""
d["total"] = d["counter"]
kanta_v1 = Kanta(filename, Data(), migrations=sys.modules[__name__])
@kanta_v1.logfmt
def resolve_user(value: str, path: str, state: dict) -> str | None:
"""Resolve user ids to names from the database state itself."""
if path != "$user" and not path.startswith("users."):
return None
return state.get("users", {}).get(value, {}).get("name")
async def main() -> None:
filename.unlink(missing_ok=True)
print("Database creation with v0 schema and basic transactions:\n")
# Open and close automatically; you can also `await kanta.open()` instead
async with kanta_v0 as kanta:
with kanta.transaction(action="create", user="userid001") as data:
data.users["userid002"] = {"name": "Bob", "role": "user"}
with kanta.transaction(action="update", user="userid001") as data:
data.users["userid002"]["role"] = "editor"
data.counter = 1
# Display-only extra string, appended after the action.
with kanta.transaction(
action="export", user="userid002", extra="extra info"
) as data:
data.counter = 2
print("\nA new data model, migrations and logfmt pretty names:\n")
async with kanta_v1 as kanta:
with kanta.transaction(
action="update", user="userid002", extra=filename.name
) as data:
data.total += 1
try:
with kanta.transaction(action="reset", user="userid001") as data:
data.total = 99
raise ValueError("simulated failure")
except ValueError:
print(
f"\nReset rolled back: {data.total=} (we can always read data without tx)\n"
)
with kanta.transaction(action="delete", user="userid002") as data:
del data.users["userid001"]
# Fake clock for deterministic timestamps
_now = datetime(2027, 1, 1, tzinfo=UTC)
@kanta_v0.clock
@kanta_v1.clock
def fake_clock() -> datetime:
global _now
_now += timedelta(hours=1)
return _now
if __name__ == "__main__":
configure_logging(debug=True)
asyncio.run(main())
+48
View File
@@ -0,0 +1,48 @@
# Bootstrap and Open Modes
When `open()` creates a brand-new database, it always writes a single bootstrap change record from the initial data object you passed to `Kanta(...)`. The simplest bootstrap is therefore the object itself — no extra code is required.
Bootstrap handlers are optional. Use them only when you need to modify the initial state at creation time, for example to seed defaults or perform expensive/external setup that should happen exactly once:
```python
kanta = Kanta("data.kantadb", Data())
@kanta.bootstrap(action="seed", user="system")
def seed_defaults(data) -> None:
data.users["admin"] = User(name="Admin")
await kanta.open()
```
You can also use `@kanta.bootstrap` with no arguments and async handlers:
```python
@kanta.bootstrap
async def bootstrap_async(data) -> None:
data.counter = 1
```
Whether or not handlers are registered, exactly one bootstrap change record is written when a new database is created. The record contains the initial object, or the state after all bootstrap handlers have run. When handlers are present:
- they run in registration order,
- bootstrap metadata (`action`, `user`, `mtime`) is taken from the last registration.
If any bootstrap handler raises, Kanta closes and removes the database file, then re-raises the error.
## Strict open mode
```python
await kanta.open(create=False)
```
With `create=False`, open fails if the database file does not exist or is empty.
## Read-only mode
Read-only mode opens an existing database without locking it or starting the background flush task. This is useful for readers that must not block the writer or modify the file:
```python
await kanta.open(readonly=True)
```
In read-only mode, records are replayed and migrations are applied in memory, but transactions and explicit flushes are rejected and the file is never created if missing.
+76 -57
View File
@@ -1,13 +1,13 @@
# Kanta Database Format and Design Principles # Kanta Database Format and Design Principles
This document describes the on-disk format and design principles of Kanta. This document describes the on-disk format and design principles of Kanta. It is intentionally focused on the current standalone package behavior.
It is intentionally focused on the current standalone package behavior.
## Core Principles ## Core Principles
1. Append-only durability 1. Append-only durability
- State changes are persisted as appended JSON lines. - State changes are persisted as appended JSON lines.
- Existing lines are never edited in place. - Existing lines are never edited in place.
- In JSON mode the history is viewable in any text editor, and a manual rollback to a prior state is possible by removing final lines (one per transaction).
2. Differential persistence 2. Differential persistence
- Kanta stores diffs (patches), not full state, for normal writes. - Kanta stores diffs (patches), not full state, for normal writes.
@@ -28,8 +28,7 @@ It is intentionally focused on the current standalone package behavior.
## On-Disk Record Types ## On-Disk Record Types
Kanta uses a newline-delimited stream where each line is either a change Kanta uses a newline-delimited stream where each line is either a change record or a snapshot record.
record or a snapshot record.
### Change record ### Change record
@@ -68,28 +67,24 @@ Fields:
3. Replay subsequent change records in order using patch application. 3. Replay subsequent change records in order using patch application.
4. The final replay state becomes in-memory `kanta.data`. 4. The final replay state becomes in-memory `kanta.data`.
This model provides fast startup for large logs while retaining append-only This model provides fast startup for large logs while retaining append-only history.
history.
## Serialization Semantics ## Serialization Semantics
- In-memory data is defined by an application `msgspec.Struct` type. - In-memory data is defined by an application `msgspec.Struct` type.
- Kanta round-trips through plain builtins for persistence and diffing. - Kanta round-trips through plain builtins for persistence and diffing.
- Dict keys are serialized as strings (`str_keys=True`) for stable JSON form. - Dict keys are serialized as strings (`str_keys=True`) for stable JSON form.
- Normalization changes introduced by struct decode/encode are logged as - Normalization changes introduced by struct decode/encode are logged together with migrations as `migrate:vN`, or as `migrate:msgspec` when no migration ran but normalization still produces a diff.
`migrate:msgspec` when they produce a diff.
## Transaction Semantics ## Transaction Semantics
- `kanta.transaction(action=...)` captures a pre-transaction snapshot dict. - `kanta.transaction(action=...)` captures a pre-transaction snapshot dict.
- By default a transaction updates the modification time `m` to the current UTC - By default a transaction updates the modification time `m` to the current UTC time.
time.
- `mtime=True|False|datetime` controls the modification time `m`: - `mtime=True|False|datetime` controls the modification time `m`:
- `True` (default) sets `m` to the current UTC time. - `True` (default) sets `m` to the current UTC time.
- `False` omits `m`, leaving the previous modification time in effect. - `False` omits `m`, leaving the previous modification time in effect.
- A `datetime` sets `m` to that explicit value. - A `datetime` sets `m` to that explicit value.
- System operations such as `migrate:msgspec` use `mtime=False` so they are not - System operations such as `migrate:msgspec` use `mtime=False` so they are not considered modifications and do not advance `m`.
considered modifications and do not advance `m`.
- On success: - On success:
- compute diff between previous builtins and current builtins, - compute diff between previous builtins and current builtins,
- queue a `ChangeRecord` if non-empty, - queue a `ChangeRecord` if non-empty,
@@ -102,9 +97,7 @@ Nested transactions are rejected.
## Modification Time ## Modification Time
`kanta.mtime` exposes the last modification time carried forward from change `kanta.mtime` exposes the last modification time carried forward from change records. It is updated by normal transactions and preserved across snapshots and reloads, while system operations such as migrations leave it unchanged.
records. It is updated by normal transactions and preserved across snapshots and
reloads, while system operations such as migrations leave it unchanged.
## Flush and Lifecycle ## Flush and Lifecycle
@@ -118,73 +111,101 @@ reloads, while system operations such as migrations leave it unchanged.
- `await kanta.open()` (default) creates the database file if missing. - `await kanta.open()` (default) creates the database file if missing.
- `await kanta.open(create=False)` fails when the file is missing or empty. - `await kanta.open(create=False)` fails when the file is missing or empty.
- `await kanta.open(readonly=True)` opens an existing database read-only.
- The file is opened without acquiring a lock and without a background flush task.
- Existing records are replayed and migrations are still applied in memory.
- Transactions and explicit flushes are rejected.
- The file is never created if missing.
### Callbacks ### Callbacks
All callbacks are registered via decorators and receive arguments by their All callbacks are registered via decorators and receive arguments by their annotation types. Parameters without a supported annotation are only allowed when they have a default value.
annotation types. Parameters without a supported annotation are only allowed
when they have a default value.
#### Bootstrap Callbacks #### Bootstrap Callbacks
- Bootstrap callbacks run during `open()` when the database is empty. - When `open()` creates a new database, it always writes a single bootstrap `ChangeRecord`.
- The simplest bootstrap is the initial data object passed to `Kanta(...)`; bootstrap callbacks are optional and only needed when you want to modify or enrich that object at creation time.
- Register callbacks via: - Register callbacks via:
- `@kanta.bootstrap` - `@kanta.bootstrap`
- `@kanta.bootstrap(action=..., user=..., mtime=...)` - `@kanta.bootstrap(action=..., user=..., mtime=...)`
- Bootstrap callbacks may be sync or async. The live root data object is - Bootstrap callbacks may be sync or async. The live root data object is injected by annotating a parameter with the struct type passed to `Kanta`, and the `Kanta` instance itself can be injected by annotating a parameter with `Kanta`.
injected by annotating a parameter with the struct type passed to `Kanta`,
and the `Kanta` instance itself can be injected by annotating a parameter
with `Kanta`.
- Multiple bootstrap callbacks are supported: - Multiple bootstrap callbacks are supported:
- callbacks execute in registration order, - callbacks execute in registration order,
- exactly one bootstrap `ChangeRecord` is queued, - exactly one bootstrap `ChangeRecord` is queued,
- bootstrap metadata (`action`, `user`, `mtime`) is taken from the last - bootstrap metadata (`action`, `user`, `mtime`) is taken from the last callback registration.
callback registration. - If no bootstrap callbacks are registered, the bootstrap record still uses `action="bootstrap"` and contains the initial data object.
- If any bootstrap callback raises, Kanta closes and removes the database file, - If any bootstrap callback raises, Kanta closes and removes the database file, then re-raises the exception.
then re-raises the exception.
#### Fatal Error Handlers #### Fatal Error Handlers
- Fatal background persistence errors can be handled with `@kanta.fatal_error`. - Fatal background persistence errors can be handled with `@kanta.fatal_error`.
- Handlers may be sync or async. The `DatabaseError` is injected by annotating - Handlers may be sync or async. The `DatabaseError` is injected by annotating a parameter with `DatabaseError`; `Kanta` may also be injected.
a parameter with `DatabaseError`; `Kanta` may also be injected. - Multiple handlers are supported and invoked in registration order. A failing handler is logged and does not prevent subsequent handlers from running.
- Multiple handlers are supported and invoked in registration order. A failing
handler is logged and does not prevent subsequent handlers from running. #### Clock
- `@kanta.clock` registers a callback `() -> datetime` that replaces the default UTC clock. Its value is used for all record timestamps (`ts`, and `m` when `mtime` is `True`) and for snapshot timestamps.
- The clock is only read when a timestamp is actually produced; no-op transactions and skipped snapshot checks do not read it.
- Register before `open()` so that bootstrap and migration records use the custom clock as well. This is mainly useful for tests and reproducible demos.
#### Transaction Log Formatting #### Transaction Log Formatting
- Logfmt callbacks prettify identifiers in the change log and are registered with - Logfmt callbacks prettify identifiers in the change log and are registered with `@kanta.logfmt`.
`@kanta.logfmt`. - A logfmt callback is called for every value Kanta renders: diff values, path components, and the transaction `user`. It receives the value as its first parameter and optionally a `path: str` parameter with the dot-notation path to the value. The special path `"$user"` is used when rendering the transaction actor.
- A logfmt callback is called for every value Kanta renders: diff values, path - The callback returns `str | None`: a string replaces the default rendering, while `None` means "fall through to the next formatter".
components, and the transaction `user`. It receives the value as its first - State dicts are injected by parameter name or annotation tag, which share the same vocabulary: `prev` receives the previous state dict and `state` the current one. Matching by name ignores the annotation entirely. The `DictPrev`/`DictState` aliases (`Annotated[dict, "prev"]` / `Annotated[dict, "state"]`) work under any parameter name, and a tag takes precedence over the name. `DictPre` and `DictPost` are kept as aliases of `DictPrev` and `DictState`. The `Kanta` instance can also be injected.
parameter and optionally a `path: str` parameter with the dot-notation path - Alternatively, a logfmt callback can be a class inheriting from `LogFmt`; the framework instantiates it with the state dicts and calls its `resolve(value, path) -> str | None` method.
to the value. The special path `"$user"` is used when rendering the - Multiple logfmt callbacks are stacked in registration order; the first callback to return a non-`None` result wins. If none handle a value, Kanta falls back to its default formatting.
transaction actor, replacing the old `user_display` parameter.
- The callback returns `str | None`: a string replaces the default rendering,
while `None` means "fall through to the next formatter".
- State dicts can be injected via `DictPre` (`Annotated[dict, "pre"]`)
and `DictPost` (`Annotated[dict, "post"]`); the `Kanta` instance can also be
injected.
- Alternatively, a logfmt callback can be a class inheriting from `LogFmt`; the
framework instantiates it with the state dicts and calls its
`resolve(value, path) -> str | None` method.
- Multiple logfmt callbacks are stacked in registration order; the first
callback to return a non-`None` result wins. If none handle a value, Kanta
falls back to its default formatting.
The decorator accepts an optional ``path`` so the callback only runs for The decorator accepts an optional ``path`` so the callback only runs for values at that exact path:
values at that exact path:
```python ```python
@kanta.logfmt(path="$user") @kanta.logfmt(path="$user")
def resolve_user(value: str, current: DictPost) -> str | None: def resolve_user(value: str, state: dict) -> str | None:
return current.get("users", {}).get(value, {}).get("name") return state.get("users", {}).get(value, {}).get("name")
@kanta.logfmt(path="users.uuid-1") @kanta.logfmt(path="users.uuid-1")
def resolve_user_key(value: str) -> str | None: def resolve_user_key(value: str) -> str | None:
return names_by_id.get(value) return names_by_id.get(value)
``` ```
#### Transaction Log Headers
- By default a transaction is logged with an `action by user` header followed by the diff lines. Added paths are colored green, deleted paths red. ANSI color codes are stripped after formatting when the standard error stream does not support color: `NO_COLOR` disables colors, `FORCE_COLOR` forces them, otherwise a tty check and a journald (`JOURNAL_STREAM`) check decide. The CLI (`python -m kanta`) strips its output the same way.
- `kanta.transaction(..., extra=...)` accepts a display-only value that is shown after the action in the header. Anything other than `None` is printed str-converted (colored by Kanta), unless a custom logemit handler does something else with it; it is never persisted in the `ChangeRecord`.
- `kanta.transaction(..., logdiff=False)` skips building and printing the diff body and logs only the header, which is useful for large or noisy changesets. Diff output can also be disabled globally with `configure_logging(diff=False)`; diff lines are emitted on the `kanta.transaction.diff` child logger so applications can route or silence them separately from the headers.
- The event loggers `kanta.bootstrap`, `kanta.migration` and `kanta.transaction` are configured at import time (via `configure_logging()`, callable again to change the toggles): a plain stderr handler with no prefix and `propagate = False`, since Kanta renders this output itself. No levels are set, so they inherit the effective root level — a framework switching root between INFO in development and WARNING in production governs Kanta output too. Operational diagnostics (integrity errors, flush failures, rotation notes) use the plain `kanta` logger instead, propagating to the root logger and following the application's normal logging configuration.
#### Log Emitters
Every change-related message Kanta emits (transaction/bootstrap/migration changes, file created/opened lines, migration summaries, aborted transactions) is described by a `kanta.logging.LogEvent` and dispatched through `kanta.logging.emit_event`. Kanta's own output goes through the same mechanism: when no `logemit` callback handles an event, `kanta.logging.default_emit` renders it with the built-in formatting.
- A `LogEvent` carries the event `kind` (`"change"`, `"created"`, `"opened"`, `"migrated"`, `"aborted"`), the preferred `logger` and `level`, the `kanta` instance, and all relevant state: `action`, `user`, `extra`, `error` (for aborted transactions), `diff`, `previous`/`current` state dicts, the built `logfmt` chain, and version info for migration events. Application-specific context (e.g. a connection id) can be stored in `kanta.ctx` — a user-writable namespace — and read back in callbacks as `event.kanta.ctx`, which also covers creation/bootstrap events.
- The built-in formatting is assembled from standard blocks that custom emitters can reuse as-is or replace piecemeal:
- `event.header` — a lazy property producing the default one-line header for any kind: `<action>[ <extra>][ by <user>]` for changes, `<action>[ <extra>][ by <user>] transaction aborted: <error>` for aborts, and the `🛢️ <file> created|opened|migrated ...` summaries. It is settable: assign `event.header = ...` and return truthy to restyle the header while keeping the default diff routing.
- `event.diff_lines` — a lazy property producing the pretty diff body for change events (built only if accessed).
- `default_emit` itself is just `header` plus the `diff_lines` routing.
- `@kanta.logemit` registers a callback receiving the event. The callback decides what is logged and where: it may log one or more messages on `event.logger`, log somewhere else, or nothing at all. A falsy return value marks the event handled and stops the chain; a truthy return value passes the event — possibly modified — to the next registered callback. When all callbacks pass, `default_emit` renders the event; a callback may also call `default_emit(event)` itself to delegate events it does not customize. Operational diagnostics (integrity errors, background flush failures) do not go through this mechanism.
- Logging never breaks functionality: a crashing `logemit` callback is reported with `logger.exception` and the event falls back to the built-in formatting; if the built-in formatting itself fails, the error is reported and swallowed. The same applies to `logfmt` callbacks (a failing one is treated as a fall-through) and `logmigr` callbacks.
```python
@kanta.logemit
def emit(ev: LogEvent):
if ev.kind != "change":
return default_emit(ev) # delegate, no chaining needed
# Restyle the header; default_emit keeps routing the diff body.
ev.header = str(Line().user(ev.user or "-", width=20)(" ").action(ev.action))
return True
```
#### Terminal Formatting Helpers
- `kanta.tty` provides the building blocks used by Kanta's own rendering:
- `colors`: the mutable color palette. Colors are bare SGR parameter strings (e.g. `"1;34"`, `"38;5;226"`) without escape framing. Attributes are read at render time, so assignments (`colors.action = "36"`) and additions (`colors.session = "38;5;226"`) take effect immediately.
- `Line`: builds a terminal string part by part. Calling it appends content (`str`-converted); `.<colorname>` arms a palette color for the next call only, and the reset is folded into a single escape sequence with whatever color comes next. `width=`/`align=` pad by display width; `str(line)` finishes the line and restores default colors.
- `strip_ansi`, `displaywidth` (wide chars and emoji count correctly) and `pad` for working with pre-colored strings.
- `use_color(stream)`: the color-support test used by Kanta's own output — honors `NO_COLOR`/`FORCE_COLOR`, then `stream.isatty()`, then the journald `JOURNAL_STREAM` device/inode match.
## Migrations ## Migrations
- Migration source is configured on `Kanta(...)` via `migrations=`. - Migration source is configured on `Kanta(...)` via `migrations=`.
@@ -195,8 +216,6 @@ def resolve_user_key(value: str) -> str | None:
## Safety Invariants ## Safety Invariants
- Any detected out-of-transaction mutation is treated as a fatal consistency - Any detected out-of-transaction mutation is treated as a fatal consistency violation.
violation.
- Flush failures mark the instance as failed and trigger shutdown behavior. - Flush failures mark the instance as failed and trigger shutdown behavior.
- Object identity of `kanta.data` is preserved across rollback when possible, - Object identity of `kanta.data` is preserved across rollback when possible, minimizing stale-reference hazards for callers.
minimizing stale-reference hazards for callers.
+14
View File
@@ -0,0 +1,14 @@
# Fatal Error Handlers
Fatal background write errors can be observed with a decorator:
```python
import os
import signal
@kanta.fatal_error
async def on_fatal(err):
os.kill(os.getpid(), signal.SIGTERM) # Die
```
Multiple fatal handlers are supported and run in registration order.
+12
View File
@@ -0,0 +1,12 @@
# Migrations
Adding or removing a field and other such simple operations are automatic, but when the time comes to really change your data model, implement a `migrate_v1` function that converts your old data to the new form. This works on plain built-in dict and other types, to avoid needing to preserve old versions of your structs.
Pass a module (or import path) containing `migrate_vN` functions:
```python
kanta = Kanta("data.kantadb", Data(), migrations="myapp.migrations")
await kanta.open()
```
Kanta tracks migration version metadata automatically, and fast forwards your database to current version by running all the migrations needed while opening the database.
+70
View File
@@ -0,0 +1,70 @@
# Database Rotation
Kanta can bound the on-disk history of a database to a configurable retention window (e.g. the last 30 days) by *rotating* the database file: the aged-out content is copied to a timestamped sibling file and the main file is truncated and rewritten in place with only the retained history plus fresh snapshots. Normal operation stays append-only under the exclusive lock; rotation is the only operation that rewrites the file.
## Configuration
Rotation is enabled with a keyword option on `Kanta(...)`:
- `retention: timedelta | int | None = None` — history window to keep; a plain `int` is interpreted as a number of days. `None` (default) disables rotation entirely.
Rotation uses the same clock as record timestamps, so a custom `@kanta.clock` callback controls it as well (useful in tests).
## When rotation runs
Rotation happens inside `Kanta.open()`, after acquiring the exclusive lock and before replay. At that point the file is quiescent: no background flush loop is running yet and no records are in flight. Rotation therefore never races with the background writer, snapshot requests, or migration snapshot writes.
Consequence: a database that is never reopened never rotates. For long-running services, rotation takes effect on the next restart.
## Rotated file naming
The history that aged out is preserved at:
```
{stem}@{ISO-8601 timestamp}.kantadb
```
- `{stem}` is the original filename with its extension stripped (`Path(filename).stem`), so databases named `data`, `data.kantadb`, or `data.db` all rotate to `data@….kantadb`.
- The timestamp is the `ts` of the last record dropped by the rotation (the leading snapshot of the rewritten main file carries the same `ts`), not the rotation time — the name tells you exactly which point in history the rotated file ends at. It is rendered in ISO 8601 basic format at second precision (e.g. `20260902T143000Z`); the exact microsecond timestamp of the cutoff remains available inside the file as the `ts` of its final line and of the leading snapshot of the new file.
- Rotated files live in the same directory as the main file.
- On collision (a rotated file with the same name already exists), an incrementing suffix is inserted before the extension (`data@….1.kantadb`, `data@….2.kantadb`, …) rather than overwriting.
- Rotated files are never deleted by rotation.
## Rotation algorithm
Let `cutoff = now - retention`. All planning happens on the in-memory bytes of the database file already read by open; no second disk read is needed.
1. **Eligibility.** Rotation is skipped when there is nothing to do: when the file contains no change records older than `cutoff` (the retention window already covers all history), or when the file contains no change records at all (a snapshot-only file is treated as already fully rotated and never re-rotated).
2. **Replay base.** The base is the newest snapshot whose `ts <= cutoff`, falling back to the start of file when no such snapshot exists. Starting at the most recent snapshot would silently drop history that must be retained.
3. **Replay and validate.** The file is replayed from the base forward to the end. Records with `ts < cutoff` are applied to the replay (they are needed to reach the cutoff state) but not retained in the output. At every snapshot encountered after the base, the replayed state is validated against the snapshot state; a mismatch means the history is corrupt, and rotation is aborted with a `DatabaseError`, leaving the original file untouched. The byte offset just after the last record with `ts < cutoff` (frame-boundary aligned) is remembered as `cutoff_end`.
4. **Copy the original aside.** The main file is copied with `shutil.copy2` directly to its final `{stem}@{ts}.kantadb` name. On filesystems with copy-on-write this performs a cheap reflink copy. A failure here aborts rotation with the original file intact.
5. **Rewrite the main file in place.** On the locked file descriptor the content is replaced (seek 0, truncate, write, fsync) with, in order:
1. A **snapshot of the state at the cutoff**, stamped with the schema version and modification time in effect at the cutoff. Its `ts` is the `ts` of the last pre-cutoff record — the same timestamp used in the rotated filename. This snapshot is the new replay base and carries the version forward so migrations are not re-run; it is always written.
2. The retained change records (`ts >= cutoff`), recreated record by record — no internal snapshots are carried over.
3. A **final snapshot** of the state after the last retained record, written only when enough changes were retained to warrant one (the same policy as regular snapshot writes). If no records survived the cutoff, the new file consists of the single leading snapshot and nothing else — the steady state for databases whose history has fully aged out.
6. **Trim the rotated copy.** The rotated file is truncated to `cutoff_end` bytes, so it contains only the dropped history and does not duplicate the records retained in the main file. The cut is at a frame boundary, so the rotated file remains a valid, replayable database on its own. This happens only after step 5's fsync, so until then the rotated file still holds the complete original content as a crash-recovery anchor.
7. **Continue normal open.** Replay and migrations proceed on the same locked file. Because the leading snapshot carries the current version, migrations run exactly as they would have against the old content.
Failure rule: any error before step 5 leaves the main file byte-identical (only an extra copy exists). A crash during step 5 may leave the main file torn, but the rotated copy still holds the complete original content — recovery is copying it back. After step 6 the split is complete and both files are consistent.
## Design notes
**In-place rewrite, not rename-and-recreate.** The writer holds an exclusive `flock` on the file from `open()` until `close()`, and `flock` is attached to the open file description (inode), not the path. Renaming the locked file away and creating a fresh file at the main path would open a race: between the rename and the creation, a second instance could open the missing path with `O_CREAT`, acquire its own lock on the fresh inode, and bootstrap a divergent database. On Windows, renaming the locked file would fail outright (the database is opened with `FILE_SHARE_READ` only). Rotation therefore never renames or unlinks the main file and never releases its lock; a second instance opening the path at any moment gets either the old content or the new, and never its own lock. The only primitive this requires is `LockedFile.replace_content()` (seek 0, truncate, write, fsync).
**Rewrite (re-frame), not verbatim copy**, for all records written to the main file. `BinFramer` checksums are offset-keyed (the checksum includes the absolute `record_offset`), so a verbatim byte copy to a new offset would be unreadable; binary records are re-framed at their new offsets. Rewriting also normalizes encoding drift and lets rotation drop the redundant intermediate snapshots the original file accumulated. The rotated copy is the one place where verbatim bytes are used — a raw `copy2` plus a frame-aligned tail truncation — which is safe precisely because it preserves original offsets: the truncated prefix keeps every frame at its original `record_offset`, so binary checksums stay valid.
## Integrity guarantees
- Rotation runs under the exclusive lock, before the background writer starts.
- The main path is never renamed, unlinked, or unlocked during rotation; no bootstrap race with a second instance is possible.
- Replay is validated against every snapshot in range; any validation failure aborts rotation with the original file intact.
- A leading cutoff snapshot (ts = last pre-cutoff record, schema version at the cutoff) is always written; a final snapshot is written only when warranted by retained changes.
- The full original content sits at `{stem}@{ts}.kantadb` before the main file is touched, and is only trimmed to the dropped-history prefix after the rewritten main file is fsynced.
- Rotated files are never deleted by rotation.
- Files with no change records (already reduced to a snapshot) are never re-rotated.
+96
View File
@@ -0,0 +1,96 @@
# Usage Patterns: Opening and Data Handling
This document covers the basics of opening a database and working with the live data object: who owns it, where initial data comes from, and the lifecycle patterns Kanta supports.
## The data object belongs to you
You pass the root state object to `Kanta(...)`, and Kanta never replaces it with a new instance. Replay, migrations, transaction rollbacks — all restore or mutate the object's contents in place, preserving its identity. This means you may hold external references to the object (or to parts of it) and they stay valid for the lifetime of the `Kanta` instance:
```python
data = Data()
kanta = Kanta("data.kantadb", data)
await kanta.open()
assert kanta.data is data # always the same object
```
Access the state as `kanta.data`, via your own reference, or both — they are the same object. Mutations must happen inside a transaction (see below); Kanta treats any detected out-of-transaction mutation as a fatal consistency violation.
Note that replacing the whole object is also possible (`kanta.data = Data()` has a setter), but then previously held references point at the old object — prefer in-place mutation.
## Where initial data comes from
The object passed to `Kanta(...)` seeds the database: when `open()` creates a brand-new (missing or empty) file, it writes a single bootstrap change record containing that object's contents, optionally modified by `@kanta.bootstrap` handlers (see [Bootstrap and open modes](bootstrap.md)).
When the file already exists, the constructor argument is *not* used as state — replay rebuilds the contents of `self.data` in place from the stored records, and the argument only defines the struct type. Consequence: if you close an instance, delete the file, and `open()` again, the new database is bootstrapped from the object's *current* contents — the old database's final state, not its original initial values. For a genuinely fresh start, construct a new data object (or reset the fields in a bootstrap handler).
## Lifecycle patterns
### Context manager (single database, scoped lifetime)
```python
async with Kanta("data.kantadb", Data()) as kanta:
with kanta.transaction(action="create_user") as data:
data.users[user_id] = User(name="Alice")
```
`async with` guarantees `open()`/`close()` pairing: the final flush happens on exit even on errors. Both `as` bindings are pure shortcuts — `async with Kanta(...) as kanta` binds the `Kanta` instance itself, and `with kanta.transaction(...) as data` binds exactly `kanta.data`. Use them or don't:
```python
async with Kanta("data.kantadb", Data()) as db:
with db.transaction(action="rename"):
db.data.users[user_id].name = "Bob" # same object as `data` above
```
### Module-level global (typical application state)
When the database lives as long as the process, define it once and open/close at application startup and shutdown:
```python
kanta = Kanta("data.kantadb", Data())
data = kanta.data # optional: your own direct reference
async def startup() -> None:
await kanta.open()
async def shutdown() -> None:
await kanta.close()
async def create_user(name: str) -> None:
with kanta.transaction(action="create_user") as d:
d.users[uuid7()] = User(name=name)
```
Transactions are synchronous context managers, so no `await` is needed per operation; the background task flushes queued changes periodically.
### Dynamically created databases (per-project, per-tenant, …)
Nothing requires module-level definitions. An app managing many databases simply constructs instances on demand and tracks them itself:
```python
class ProjectStore:
def __init__(self) -> None:
self.projects: dict[str, Kanta[Data]] = {}
async def get(self, project_id: str) -> Kanta[Data]:
kanta = self.projects.get(project_id)
if kanta is None:
kanta = Kanta(f"projects/{project_id}.kantadb", Data())
await kanta.open()
self.projects[project_id] = kanta
return kanta
```
Remember that each open writer holds an exclusive lock on its file, so keep one `Kanta` instance per path and close instances you no longer need.
## Transactions in brief
- `with kanta.transaction(action=..., user=...) as data:` yields the live state object for mutation.
- On success, Kanta computes a diff against the pre-transaction state and queues a change record; on exception, the in-memory data is rolled back and the exception is re-raised.
- Nested transactions are rejected.
- A transaction that changes nothing queues no record.
See [On-disk format](database.md) for the full transaction semantics and [Validation](validation.md) for integrity checks that run inside each transaction.
+20
View File
@@ -0,0 +1,20 @@
# Validation
`@kanta.validate` registers an integrity validator for the database state, beyond the structural validation msgspec already performs during decoding.
```python
@kanta.validate
def check_users(data: Data) -> None:
for user in data.users.values():
if not user.name:
raise ValueError("user without a name")
```
Validators receive the live data object (and optionally the `Kanta` instance as a second annotated parameter) and must **raise an exception** when the data is inconsistent. They are not intended to modify or correct the data — only to fail.
Validators run:
- **on open** — after replay, msgspec decoding and migrations, before the database becomes usable; a failure aborts the open and releases the file,
- **after each transaction** — before the change is committed to history; a failure rolls the transaction back, so invalid state never reaches the log.
Multiple validators may be registered; they run in registration order until the first failure. Validators must be synchronous (transactions are synchronous) — async callbacks are rejected at registration time.
+15
View File
@@ -1,5 +1,20 @@
from .callbacks import DictPrev, DictState, LogFmt
from .diff import diff, patch
from .exceptions import DatabaseError
from .kanta import Kanta from .kanta import Kanta
from .logging import LogEvent, configure_logging
from .migrations import MigrationReport
__all__ = [ __all__ = [
"Kanta", "Kanta",
"DatabaseError",
"configure_logging",
"diff",
"patch",
# Callback argument types
"DictPrev",
"DictState",
"LogEvent",
"LogFmt",
"MigrationReport",
] ]
+681
View File
@@ -0,0 +1,681 @@
"""Module-level CLI for reading a kantadb file and printing its change log."""
from __future__ import annotations
import argparse
import asyncio
import contextlib
import importlib
import importlib.metadata
import importlib.util
import sys
import tempfile
from pathlib import Path
from typing import Any
import msgspec
from kanta import Kanta
from kanta.callbacks import InjectionContext, callback_error_reporter
from kanta.exceptions import DatabaseError, DataIntegrityError, ReplayError
from kanta.grep import GrepPattern, evaluate, matches_snapshot
from kanta.logging import (
LogEvent,
emit_event,
format_action_header,
format_diff,
migration_logger,
)
from kanta.replaylog import (
RangeNotFoundError,
Selection,
SnapshotEvent,
_snapshot_lines,
end_of_file,
record_change_event,
record_label,
replay_events,
scan_events,
select,
)
from kanta.serialization import Serializer, dict_to_struct, struct_to_dict
from kanta.structs import ChangeRecord, Snapshot
from kanta.tty import Line, strip_ansi, use_color
EXIT_SUCCESS = 0
EXIT_GENERIC = 1
EXIT_RANGE_ERROR = 2
EXIT_PARSE_ERROR = 10
EXIT_MIGRATION_ERROR = 20
EXIT_VALIDATION_ERROR = 21
def _print(*args: Any) -> None:
"""Print to stderr, stripping ANSI codes when the stream has no color support.
Color detection runs per call so redirected or reassigned ``sys.stderr``
(and environment changes) are honored; ANSI codes are stripped after
formatting, not by formatting differently.
"""
text = " ".join(str(arg) for arg in args)
if not use_color():
text = strip_ansi(text)
print(text, file=sys.stderr)
class _CliError(Exception):
"""A user-facing error message paired with a process exit code."""
def __init__(self, message: str, code: int = EXIT_GENERIC) -> None:
self.code = code
super().__init__(message)
def _import_dotted(path: str) -> Any:
"""Import ``module.submodule.Attr`` or a filesystem path and return the attribute."""
if _is_file_path(path):
return _import_from_file(path)
if "." not in path:
raise ValueError(f"dotted path must contain a dot: {path!r}")
module_name, attr_name = path.rsplit(".", 1)
module = importlib.import_module(module_name)
try:
return getattr(module, attr_name)
except AttributeError as exc:
raise ImportError(f"{path!r} not found in {module_name!r}") from exc
def _is_file_path(path: str) -> bool:
"""Return True if *path* looks like a filesystem path rather than a dotted name."""
return "/" in path or "\\" in path or ":" in path
def _import_from_file(path: str) -> Any:
"""Import a module or attribute from a filesystem path.
*path* may be ``path/to/file.py`` (returns the module) or
``path/to/file.py:symbol`` (returns ``symbol`` from the module).
"""
if ":" in path:
file_path, symbol = path.rsplit(":", 1)
else:
file_path, symbol = path, None
file_path = Path(file_path).resolve()
if not file_path.exists():
raise ImportError(f"{file_path!r} not found")
if not file_path.is_file():
raise ImportError(f"{file_path!r} is not a file")
module_name = f"_kanta_cli_{file_path.stem}_{file_path.stat().st_ino}"
spec = importlib.util.spec_from_file_location(module_name, file_path)
if spec is None or spec.loader is None:
raise ImportError(f"cannot load {file_path!r}")
module = importlib.util.module_from_spec(spec)
file_dir = str(file_path.parent)
added_dir = False
if file_dir not in sys.path:
sys.path.insert(0, file_dir)
added_dir = True
try:
sys.modules[module_name] = module
spec.loader.exec_module(module)
finally:
if added_dir:
sys.path.remove(file_dir)
if symbol is None:
return module
try:
return getattr(module, symbol)
except AttributeError as exc:
raise ImportError(f"{symbol!r} not found in {file_path!r}") from exc
def _import_kanta_object(path: str) -> Any:
"""Import a Kanta object by module or filesystem path.
If ``path`` names an importable module, look up an object named
``kanta`` in it; otherwise treat ``path`` as ``module.attr`` or
``path/to/file.py[:kanta]`` referring directly to the object.
"""
if _is_file_path(path):
if ":" in path:
return _import_from_file(path)
module = _import_from_file(path)
try:
return getattr(module, "kanta")
except AttributeError as exc:
raise ImportError(f"no 'kanta' object found in {path!r}") from exc
try:
spec = importlib.util.find_spec(path)
except ImportError:
spec = None
if spec is not None:
module = importlib.import_module(path)
try:
return getattr(module, "kanta")
except AttributeError as exc:
raise ImportError(f"no 'kanta' object found in module {path!r}") from exc
return _import_dotted(path)
def _format_ts(dt) -> str:
"""Return a local-looking timestamp without a timezone offset or microseconds."""
return dt.replace(tzinfo=None, microsecond=0).isoformat(sep=" ")
def _package_version() -> str:
"""Return the installed package version, or ``"unknown"`` from a source tree."""
try:
return importlib.metadata.version("kanta")
except importlib.metadata.PackageNotFoundError:
return "unknown"
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
prog="kanta",
description=(
f"kanta {_package_version()} - read a kantadb file and print each"
" change record to the console."
),
)
parser.add_argument(
"--version",
action="version",
version=f"%(prog)s {_package_version()}",
)
parser.add_argument(
"file",
help="Path to the kantadb file, or '-' to read from stdin.",
)
parser.add_argument(
"-d",
"--data",
metavar="MOD",
help=(
"Dotted path or filesystem path to the root data type."
" Examples: myapp.models.Data, myapp/models.py:Data."
),
)
parser.add_argument(
"-m",
"--migrations",
metavar="MOD",
help=(
"Dotted path or filesystem path to the migrations module."
" Examples: myapp.migrations, myapp/migrations.py."
),
)
parser.add_argument(
"-k",
"--kanta",
metavar="MOD",
help=(
"Module path or filesystem path to an existing Kanta object to use."
" Either a module containing an object named 'kanta' (e.g. myapp.db),"
" a dotted path to the object (e.g. myapp.db.kanta), or a file path"
" (e.g. myapp/db.py or myapp/db.py:kanta). Its type, migrations, and"
" logfmt/logemit callbacks are used. Cannot be combined with -d or -m."
),
)
parser.add_argument(
"-o",
"--output",
help="Write the final replayed state as JSON to this file, or '-' for stdout.",
)
parser.add_argument(
"-q",
"--quiet",
action="store_true",
help="Suppress normal change/snapshot logs; only print warnings and errors.",
)
parser.add_argument(
"-r",
"--range",
help=(
"Python-style range to process. Units: plain number = change index,"
" lN = line number, sN = snapshot, vN = version. Negative snapshot"
" values count from the end (s-1 is the last snapshot). Use ':' for"
" half-open ranges and '..' for inclusive end ranges. Examples:"
" '2:5', '2..5', 'l10:l20', 's1:s3', 'v0:v2', 's-1:', ':-1', '-1'."
),
)
parser.add_argument(
"-g",
"--grep",
action="append",
metavar="PATTERN",
help=(
"Print only change records matching PATTERN, structurally and"
" case-insensitively; matched regions get a yellow background,"
" and on a match the whole record is printed, not just the"
" matching line. Repeatable: every pattern must match somewhere"
" in the same record, but different patterns may match different"
" lines of it. A bare pattern matches the action or user as a"
" substring, a dotted path by element (each element in full,"
" unless it uses wildcards: 'users' matches 'users' anywhere but"
" not 'foousers', 'us*' does), or a value (strings by substring,"
" other values in full: 'true' matches a boolean, 'tru' does"
" not). The 'path=value' form requires the path and the value"
" to match within the same change line; use '=value' or 'path='"
" to match values or paths only. With -k, logfmt-prettified"
" values and users match alongside the raw ones. Examples:"
" --grep alice, --grep 'users.*.email', --grep"
" 'users.alice.admin=true', --grep create_user --grep"
" '@example.com'."
),
)
args = parser.parse_args(argv)
if args.kanta and (args.data or args.migrations):
parser.error("-k/--kanta cannot be used together with -d or -m")
return args
def _print_change_log(
label: str,
record: ChangeRecord,
previous: dict[str, Any],
current: dict[str, Any],
kanta: Kanta[Any],
highlight: Any = None,
) -> None:
"""Log a single change record to stderr.
The record is dispatched as a :class:`LogEvent` through the Kanta
object's logemit handlers; the CLI's own rendering (with the ``l<N>``
label and timestamp) is the fallback when no handler claims the event.
``highlight`` is an optional :class:`kanta.grep.GrepHighlighter` with
the record's matched regions, applied to the fallback rendering.
"""
event = record_change_event(record, previous, current, kanta)
def render(ev) -> None:
ts = _format_ts(record.ts)
if highlight is not None:
header = format_action_header(
ev.action or "", ev.user, ev.extra, highlight=highlight
)
lines = format_diff(ev.diff, ev.previous, ev.logfmt, highlight=highlight)
else:
header, lines = ev.header, ev.diff_lines
if not lines:
_print(f"{label} {ts} {header}")
elif len(lines) == 1:
_print(f"{label} {ts} {header}{lines[0]}")
else:
_print(f"{label} {ts} {header}")
for line in lines:
_print(line)
_print()
emit_event(
event,
kanta._impl.callback_registry.logemit_handlers,
fallback=render,
)
def _format_size(n: int) -> str:
"""Return a human-readable byte size."""
if n < 1024:
return f"{n} B"
if n < 1024 * 1024:
return f"{n / 1024:.1f} kB"
return f"{n / (1024 * 1024):.1f} MB"
def _find_venv_site_packages(start: Path) -> list[Path]:
"""Return site-packages dirs of ``.venv`` directories from *start* to parents."""
py_dir = f"python{sys.version_info.major}.{sys.version_info.minor}"
found: list[Path] = []
for parent in [start, *start.parents]:
venv = parent / ".venv"
if not venv.is_dir():
continue
site_packages = venv / "lib" / py_dir / "site-packages"
if site_packages.is_dir():
found.append(site_packages)
continue
# Windows layout
win_site = venv / "Lib" / "site-packages"
if win_site.is_dir():
found.append(win_site)
return found
@contextlib.contextmanager
def _extra_import_paths():
"""Temporarily add current dir and nearby venv site-packages to ``sys.path``.
The current directory is inserted first, then local ``.venv`` site-packages,
then any parent ``.venv`` site-packages. Only paths that were not already
present are added, and only those added paths are removed on exit.
"""
paths_to_add = [str(Path.cwd())]
paths_to_add.extend(str(p) for p in _find_venv_site_packages(Path.cwd()))
added: list[str] = []
for path in reversed(paths_to_add):
if path not in sys.path:
sys.path.insert(0, path)
added.append(path)
try:
yield
finally:
for path in added:
if path in sys.path:
sys.path.remove(path)
def _print_snapshot_indicator(
label: str,
snap: Snapshot,
index: int,
serializer: Serializer,
) -> None:
"""Print a snapshot indicator line to stderr.
``snapshot s<N>`` is rendered in bright white; the version, optional mtime
and data size are printed in normal and dark colors respectively.
"""
ts = _format_ts(snap.ts)
line = Line().snapshot("snapshot").snapshot(f" s{index}")
line.target(f" v{snap.v}")
if snap.m is not None:
line.target(f" {_format_ts(snap.m)}")
size = len(serializer.encode(snap.state))
line.path_prefix(f" {_format_size(size)}")
_print(f"{label} {ts} {line}")
async def _log_migration(
kanta: Kanta[Any],
filename: Path,
result,
previous_version: int,
quiet: bool,
) -> None:
"""Log an applied migration through the Kanta instance's callbacks.
Routes to the object's logmigr callbacks when registered (like
:meth:`KantaImpl._handle_migration_log`); otherwise emits a ``migrated``
event through its logemit handlers, falling back to a stderr line.
"""
registry = kanta._impl.callback_registry
if registry.has("logmigr"):
await registry.invoke(
"logmigr",
InjectionContext(kanta=kanta, report=result),
on_error=callback_error_reporter("logmigr"),
)
return
if quiet:
return
descriptions = [f"{m.name} ({m.description})" for m in result.applied if m.changed]
emit_event(
LogEvent(
kind="migrated",
logger=migration_logger,
kanta=kanta,
filename=str(filename),
from_version=previous_version,
to_version=result.version,
migrations=descriptions,
),
registry.logemit_handlers,
fallback=lambda ev: _print(ev.header),
)
def _get_kanta(args: argparse.Namespace, filename: Path) -> tuple[Kanta[Any], bool]:
"""Return the Kanta instance to work with, and whether the CLI owns it.
With ``-k`` the existing object is used as-is (and never closed by us);
otherwise an instance is constructed with an empty dict state.
"""
if args.kanta:
try:
obj = _import_kanta_object(args.kanta)
except (ImportError, ValueError) as exc:
raise _CliError(f"Invalid --kanta value: {exc}") from exc
if not isinstance(obj, Kanta):
raise _CliError(
f"Invalid --kanta value: {args.kanta!r} is not a Kanta object"
)
return obj, False
try:
return Kanta(filename, {}, type=dict, migrations=args.migrations), True
except Exception as exc:
if args.migrations:
raise _CliError(f"Migration error: {exc}", EXIT_MIGRATION_ERROR) from exc
raise _CliError(f"Failed to initialize database: {exc}") from exc
async def _run(args: argparse.Namespace) -> int:
cleanup_path: Path | None = None
if args.file == "-":
content = sys.stdin.buffer.read()
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".kantadb")
tmp.write(content)
tmp.close()
filename = Path(tmp.name)
cleanup_path = filename
else:
filename = Path(args.file)
if not filename.exists():
raise _CliError(f"File not found: {filename}")
content = filename.read_bytes()
data_type: type[Any] | None = None
if args.data:
with _extra_import_paths():
try:
data_type = _import_dotted(args.data)
except (ImportError, ValueError) as exc:
raise _CliError(f"Invalid --data value: {exc}") from exc
kanta: Kanta[Any] | None = None
kanta_owned = False
kanta_typed: Kanta[Any] | None = None
try:
with _extra_import_paths():
kanta, kanta_owned = _get_kanta(args, filename)
if data_type is None and args.kanta and kanta._impl.data_type is not dict:
data_type = kanta._impl.data_type
# Decode and validate the whole file into positioned events.
try:
events, change_count = scan_events(content, kanta)
except ReplayError as exc:
raise _CliError(str(exc), EXIT_PARSE_ERROR) from exc
except Exception as exc:
raise _CliError(
f"Failed to replay records from {filename}: {exc}",
EXIT_PARSE_ERROR,
) from exc
snapshot_line_to_index = {
line: idx for idx, line in enumerate(_snapshot_lines(events))
}
# Resolve -r into a line range or a single snapshot selection.
try:
selection = (
select(args.range, events, change_count)
if args.range is not None
else Selection(0, end_of_file(events))
)
except RangeNotFoundError as exc:
raise _CliError(str(exc), EXIT_RANGE_ERROR) from exc
except ValueError as exc:
raise _CliError(f"Invalid --range value: {exc}") from exc
grep_patterns = [GrepPattern.parse(p) for p in args.grep or ()]
if selection.snapshot is not None:
snap_event = selection.snapshot
state = snap_event.snap.state
version = snap_event.snap.v
if not args.quiet:
_print_snapshot_indicator(
record_label(snap_event.line_number, snap_event.record_index),
snap_event.snap,
snapshot_line_to_index[snap_event.line_number],
kanta._impl.serializer,
)
_print()
else:
# Replay up to the range end, printing logs within the range.
state = {}
version = 0
printed = False
for event, previous, current in replay_events(events, selection.end_line):
state = current
version = event.version
if event.line_number < selection.start_line or args.quiet:
continue
label = record_label(event.line_number, event.record_index)
if isinstance(event, SnapshotEvent):
if grep_patterns:
logfmt = kanta._impl.callback_registry.build_logfmt(
InjectionContext(
kanta=kanta,
previous_state=current,
current_state=current,
)
)
if not matches_snapshot(current, grep_patterns, logfmt=logfmt):
continue
_print_snapshot_indicator(
label,
event.snap,
snapshot_line_to_index[event.line_number],
kanta._impl.serializer,
)
else:
assert previous is not None
highlight = None
if grep_patterns:
# Build the same logfmt the rendering uses, so both
# raw and prettified values are matched.
logfmt = kanta._impl.callback_registry.build_logfmt(
InjectionContext(
kanta=kanta,
previous_state=previous,
current_state=current,
)
)
highlight = evaluate(
event.record, previous, grep_patterns, logfmt=logfmt
)
if highlight is None:
continue
_print_change_log(
label, event.record, previous, current, kanta, highlight
)
printed = True
if printed:
_print()
# Apply optional migrations to the range-end state.
if kanta._impl.migrations is not None:
try:
previous_version = version
result = kanta._impl.migrations.apply(state, version, kanta)
version = result.version
except Exception as exc:
raise _CliError(
f"Migration error: {exc}", EXIT_MIGRATION_ERROR
) from exc
if version != previous_version:
await _log_migration(
kanta, filename, result, previous_version, args.quiet
)
output_state: dict[str, Any]
if data_type is not None:
try:
data = dict_to_struct(
state, data_type, serializer=kanta._impl.serializer
)
except (
msgspec.ValidationError,
msgspec.DecodeError,
TypeError,
ValueError,
) as exc:
raise _CliError(
f"Validation error: {exc}", EXIT_VALIDATION_ERROR
) from exc
if args.kanta:
# The file was already fully decoded and validated above with
# the object's own serializer, and its migrations were applied
# to the state; no need to re-open through a new instance.
_print(f"{data}")
output_state = struct_to_dict(data, serializer=kanta._impl.serializer)
else:
kanta_typed = Kanta(
filename, data, type=data_type, migrations=args.migrations
)
try:
await kanta_typed.open(create=False, readonly=True, log=False)
_print(f"{data}")
except (msgspec.ValidationError, msgspec.DecodeError) as exc:
raise _CliError(
f"Validation error: {exc}", EXIT_VALIDATION_ERROR
) from exc
except DataIntegrityError as exc:
raise _CliError(f"Parse error: {exc}", EXIT_PARSE_ERROR) from exc
except DatabaseError as exc:
if not args.migrations or exc.cause_type == "ReplayError":
raise _CliError(
f"Parse error: {exc}", EXIT_PARSE_ERROR
) from exc
raise _CliError(
f"Migration error: {exc}", EXIT_MIGRATION_ERROR
) from exc
except Exception as exc:
if args.migrations:
raise _CliError(
f"Migration error: {exc}", EXIT_MIGRATION_ERROR
) from exc
raise _CliError(f"Failed to open {filename}: {exc}") from exc
output_state = kanta_typed._impl.statedict
else:
output_state = state
if args.output:
try:
out_bytes = msgspec.json.encode(output_state)
if args.output == "-":
sys.stdout.buffer.write(out_bytes)
else:
Path(args.output).write_bytes(out_bytes)
except Exception as exc: # pragma: no cover
raise _CliError(f"Failed to write output: {exc}") from exc
return EXIT_SUCCESS
finally:
if kanta_typed is not None:
await kanta_typed.close()
if kanta is not None and kanta_owned:
await kanta.close()
if cleanup_path is not None:
cleanup_path.unlink(missing_ok=True)
def main(argv: list[str] | None = None) -> int:
"""Entry point for ``python -m kanta``."""
args = _parse_args(argv)
try:
return asyncio.run(_run(args))
except _CliError as exc:
_print(exc)
return exc.code
if __name__ == "__main__":
sys.exit(main())
+197 -65
View File
@@ -1,41 +1,111 @@
"""Unified decorator-based callback registry for Kanta. """Unified decorator-based callback registry for Kanta.
Callbacks are registered once and invoked with arguments filled by their Callbacks are registered once and invoked with arguments filled from their
annotation types. Unknown arguments are only permitted when they have a parameter names (state dicts: ``prev`` / ``state``) and annotation types.
default value. Unknown arguments are only permitted when they have a default value.
Log formatters are a special case: they are called per value being rendered Log formatters are a special case: they are called per value being rendered
and receive the value plus an optional ``path`` string. They return and receive the value plus an optional ``path`` string. They return
``str | None``; ``None`` means "fall through to the next formatter". ``str | None``; ``None`` means "fall through to the next formatter".
Log emitters (``logemit``) are another special case: plain callables that
receive a :class:`kanta.logging.LogEvent` and are dispatched by
:func:`kanta.logging.emit_event`.
""" """
from __future__ import annotations from __future__ import annotations
import inspect import inspect
import logging
import types
from collections.abc import Callable from collections.abc import Callable
from dataclasses import dataclass from dataclasses import dataclass
from typing import Annotated, Any, Union, get_args, get_origin from typing import Annotated, Any, Union, get_args, get_origin
from kanta.exceptions import DatabaseError from kanta.exceptions import DatabaseError
from kanta.migrations import MigrationReport
DictPre = Annotated[dict, "pre"] DictPrev = DictPre = Annotated[dict, "prev"]
DictPost = Annotated[dict, "post"] DictState = DictPost = Annotated[dict, "state"]
# State-dict injection keys, shared by parameter names and annotation tags:
# a callback parameter named *or* tagged ``prev``/``state`` receives the
# previous or current state dict respectively. Matching by name does not
# check the annotation; an explicit tag takes precedence over the name.
_STATE_KINDS = {"prev": "previous_state", "state": "current_state"}
def _state_key(text: Any) -> str | None:
"""Return the state kind for a parameter name or annotation tag."""
return text if text in _STATE_KINDS else None
def _state_tag(ann: Any) -> str | None:
"""Return the state tag of an ``Annotated[dict, ...]`` annotation, if any."""
if get_origin(ann) is not Annotated:
return None
args = get_args(ann)
if not args or args[0] is not dict:
return None
for meta in args[1:]:
key = _state_key(meta)
if key is not None:
return key
return None
_logger = logging.getLogger("kanta")
def describe_callback(callback: Callable[..., Any]) -> str:
"""Return ``name (docstring first line)`` identifying *callback*.
Used in failure messages so a bare log line names the function that
failed, e.g. ``myformatter (Concise log formatter)``. Callables without
a ``__name__`` (partials, callable instances, ...) are described by
their type name only: less information, but never wrong information.
"""
name = getattr(callback, "__name__", None)
if not isinstance(name, str):
return type(callback).__name__
doc = inspect.getdoc(callback)
if doc:
return f"{name} ({doc.splitlines()[0]})"
return name
def callback_error_reporter(
kind: str,
) -> Callable[[Exception, Callable[..., Any]], None]:
"""Return an ``on_error`` reporter for :meth:`CallbackRegistry.invoke`.
The returned callable logs ``Kanta.<kind> <name (docstring)> failed``
with the traceback for each failing callback; invoke continues with
the rest.
"""
def _report(callback_error: Exception, callback: Callable[..., Any]) -> None:
_logger.exception("Kanta.%s %s failed", kind, describe_callback(callback))
return _report
class LogFmt: class LogFmt:
"""Base class for stateful logfmt callbacks. """Base class for stateful logfmt callbacks.
Subclasses only need to override :meth:`resolve`. The framework injects Subclasses only need to override :meth:`resolve`. The framework injects
``previous_state`` and ``current_state`` through ``__init__``. the previous and current state dicts through ``__init__`` and exposes them
as ``previous_state`` and ``state``.
""" """
def __init__( def __init__(
self, self,
previous: DictPre | None = None, prev: dict | None = None,
current: DictPost | None = None, state: dict | None = None,
) -> None: ) -> None:
self.previous_state = previous self.previous_state = prev
self.current_state = current self.state = state
self.current_state = state # deprecated alias for ``state``
def __call__(self, value: Any, path: str) -> str | None: def __call__(self, value: Any, path: str) -> str | None:
return self.resolve(value, path) return self.resolve(value, path)
@@ -58,6 +128,7 @@ class InjectionContext:
error: DatabaseError | None = None error: DatabaseError | None = None
previous_state: dict | None = None previous_state: dict | None = None
current_state: dict | None = None current_state: dict | None = None
report: MigrationReport | None = None
@dataclass @dataclass
@@ -97,8 +168,11 @@ class CallbackRegistry:
self._callbacks: dict[str, list[_CallbackRegistration]] = { self._callbacks: dict[str, list[_CallbackRegistration]] = {
"bootstrap": [], "bootstrap": [],
"fatal_error": [], "fatal_error": [],
"logmigr": [],
"validate": [],
} }
self._logfmt_callbacks: list[_LogFmtFunctionSpec | _LogFmtClassSpec] = [] self._logfmt_callbacks: list[_LogFmtFunctionSpec | _LogFmtClassSpec] = []
self._logemit_callbacks: list[Callable[..., Any]] = []
def register( def register(
self, self,
@@ -119,6 +193,14 @@ class CallbackRegistry:
) )
return callback return callback
if kind == "logemit":
if inspect.isclass(callback) or not callable(callback):
raise TypeError("logemit callbacks must be functions")
if inspect.iscoroutinefunction(callback):
raise TypeError("logemit callbacks must not be async")
self._logemit_callbacks.append(callback)
return callback
if kind not in self._callbacks: if kind not in self._callbacks:
raise ValueError(f"unknown callback kind: {kind}") raise ValueError(f"unknown callback kind: {kind}")
@@ -126,6 +208,10 @@ class CallbackRegistry:
raise TypeError(f"{kind} callbacks must be functions, not classes") raise TypeError(f"{kind} callbacks must be functions, not classes")
if not callable(callback): if not callable(callback):
raise TypeError(f"{kind} callback must be callable") raise TypeError(f"{kind} callback must be callable")
if kind == "validate" and inspect.iscoroutinefunction(callback):
raise TypeError(
"validate callbacks must not be async (transactions are synchronous)"
)
params = self._validate_function(callback, kind) params = self._validate_function(callback, kind)
is_async = inspect.iscoroutinefunction(callback) is_async = inspect.iscoroutinefunction(callback)
@@ -167,20 +253,39 @@ class CallbackRegistry:
break break
return results return results
def invoke_sync(self, kind: str, ctx: InjectionContext) -> None:
"""Invoke all sync callbacks of *kind* in order; first exception raises.
Used for ``validate`` callbacks, which run inside synchronous
transactions and therefore must not be async.
"""
for reg in self._callbacks[kind]:
kwargs = self._build_kwargs(reg.params, ctx)
reg.callback(**kwargs)
def has(self, kind: str) -> bool: def has(self, kind: str) -> bool:
"""Return True if any callback of *kind* is registered.""" """Return True if any callback of *kind* is registered."""
if kind == "logfmt": if kind == "logfmt":
return bool(self._logfmt_callbacks) return bool(self._logfmt_callbacks)
if kind == "logemit":
return bool(self._logemit_callbacks)
return bool(self._callbacks[kind]) return bool(self._callbacks[kind])
@property
def logemit_handlers(self) -> list[Callable[..., Any]]:
"""Registered logemit callbacks in registration order."""
return self._logemit_callbacks
def build_logfmt(self, ctx: InjectionContext) -> Callable[[Any, str], str | None]: def build_logfmt(self, ctx: InjectionContext) -> Callable[[Any, str], str | None]:
"""Build a chained formatter from registered logfmt callbacks.""" """Build a chained formatter from registered logfmt callbacks."""
formatters: list[tuple[Callable[[Any, str], str | None], str | None]] = [] formatters: list[
tuple[Callable[[Any, str], str | None], str | None, Callable[..., Any]]
] = []
for spec in self._logfmt_callbacks: for spec in self._logfmt_callbacks:
if isinstance(spec, _LogFmtClassSpec): if isinstance(spec, _LogFmtClassSpec):
kwargs = self._build_kwargs(spec.inject_params, ctx) kwargs = self._build_kwargs(spec.inject_params, ctx)
instance: Callable[[Any, str], str | None] = spec.cls(**kwargs) instance: Callable[[Any, str], str | None] = spec.cls(**kwargs)
formatters.append((instance, spec.path)) formatters.append((instance, spec.path, spec.cls))
else: else:
kwargs = self._build_kwargs(spec.inject_params, ctx) kwargs = self._build_kwargs(spec.inject_params, ctx)
@@ -200,13 +305,21 @@ class CallbackRegistry:
return formatter return formatter
formatters.append((make_formatter(), spec.path)) formatters.append((make_formatter(), spec.path, spec.callback))
def format_value(value: Any, path: str) -> str | None: def format_value(value: Any, path: str) -> str | None:
for fn, pattern in formatters: for fn, pattern, callback in formatters:
if pattern is not None and path != pattern: if pattern is not None and path != pattern:
continue continue
resolved = fn(value, path) try:
resolved = fn(value, path)
except Exception:
# Formatting must never break functionality; a failing
# callback is reported and treated as a fall-through.
_logger.exception(
"Kanta.logfmt %s failed", describe_callback(callback)
)
continue
if resolved is not None: if resolved is not None:
return resolved return resolved
return None return None
@@ -228,6 +341,9 @@ class CallbackRegistry:
) )
if param.annotation is inspect.Parameter.empty: if param.annotation is inspect.Parameter.empty:
if _state_key(name) is not None:
params.append((name, dict))
continue
if param.default is inspect.Parameter.empty: if param.default is inspect.Parameter.empty:
raise TypeError( raise TypeError(
f"{kind} callback {callback.__name__} has parameter " f"{kind} callback {callback.__name__} has parameter "
@@ -236,6 +352,11 @@ class CallbackRegistry:
continue continue
ann = self._resolve_raw_annotation(param.annotation, callback) ann = self._resolve_raw_annotation(param.annotation, callback)
if _state_key(name) is not None:
# The name alone selects state injection; an explicit tag
# still overrides it. The annotation is not checked.
params.append((name, ann))
continue
if not self._is_allowed(kind, ann): if not self._is_allowed(kind, ann):
if param.default is inspect.Parameter.empty: if param.default is inspect.Parameter.empty:
raise TypeError( raise TypeError(
@@ -296,6 +417,9 @@ class CallbackRegistry:
f"*args or **kwargs" f"*args or **kwargs"
) )
if param.annotation is inspect.Parameter.empty: if param.annotation is inspect.Parameter.empty:
if _state_key(name) is not None:
inject_params.append((name, dict))
continue
if param.default is inspect.Parameter.empty: if param.default is inspect.Parameter.empty:
raise TypeError( raise TypeError(
f"logfmt callback {callback.__name__} has parameter " f"logfmt callback {callback.__name__} has parameter "
@@ -304,6 +428,11 @@ class CallbackRegistry:
continue continue
ann = self._resolve_raw_annotation(param.annotation, callback) ann = self._resolve_raw_annotation(param.annotation, callback)
if _state_key(name) is not None:
# The name alone selects state injection; an explicit tag
# still overrides it. The annotation is not checked.
inject_params.append((name, ann))
continue
if name == "path" and self._unwrap_optional(ann) is str: if name == "path" and self._unwrap_optional(ann) is str:
has_path = True has_path = True
continue continue
@@ -317,17 +446,13 @@ class CallbackRegistry:
f"Allowed: str path, {self._allowed_message('logfmt')}" f"Allowed: str path, {self._allowed_message('logfmt')}"
) )
if sig.return_annotation is inspect.Signature.empty: if sig.return_annotation is not inspect.Signature.empty:
raise TypeError( return_ann = self._resolve_raw_annotation(sig.return_annotation, callback)
f"logfmt callback {callback.__name__} must annotate its " if not self._is_optional_str(return_ann):
f"return type as str | None" raise TypeError(
) f"logfmt callback {callback.__name__} must return str | None, "
return_ann = self._resolve_raw_annotation(sig.return_annotation, callback) f"got {return_ann!r}"
if not self._is_optional_str(return_ann): )
raise TypeError(
f"logfmt callback {callback.__name__} must return str | None, "
f"got {return_ann!r}"
)
return _LogFmtFunctionSpec( return _LogFmtFunctionSpec(
callback=callback, callback=callback,
@@ -363,6 +488,9 @@ class CallbackRegistry:
f"*args or **kwargs" f"*args or **kwargs"
) )
if param.annotation is inspect.Parameter.empty: if param.annotation is inspect.Parameter.empty:
if _state_key(name) is not None:
inject_params.append((name, dict))
continue
if param.default is inspect.Parameter.empty: if param.default is inspect.Parameter.empty:
raise TypeError( raise TypeError(
f"logfmt class {cls.__name__}.__init__ has parameter " f"logfmt class {cls.__name__}.__init__ has parameter "
@@ -371,6 +499,11 @@ class CallbackRegistry:
continue continue
ann = self._resolve_raw_annotation(param.annotation, cls.__init__) ann = self._resolve_raw_annotation(param.annotation, cls.__init__)
if _state_key(name) is not None:
# The name alone selects state injection; an explicit tag
# still overrides it. The annotation is not checked.
inject_params.append((name, ann))
continue
if self._is_allowed("logfmt", ann): if self._is_allowed("logfmt", ann):
inject_params.append((name, ann)) inject_params.append((name, ann))
continue continue
@@ -416,19 +549,15 @@ class CallbackRegistry:
f"logfmt class {cls.__name__}.resolve must accept a 'path: str' parameter" f"logfmt class {cls.__name__}.resolve must accept a 'path: str' parameter"
) )
if resolve_sig.return_annotation is inspect.Signature.empty: if resolve_sig.return_annotation is not inspect.Signature.empty:
raise TypeError( return_ann = self._resolve_raw_annotation(
f"logfmt class {cls.__name__}.resolve must annotate its " resolve_sig.return_annotation, resolve
f"return type as str | None"
)
return_ann = self._resolve_raw_annotation(
resolve_sig.return_annotation, resolve
)
if not self._is_optional_str(return_ann):
raise TypeError(
f"logfmt class {cls.__name__}.resolve must return str | None, "
f"got {return_ann!r}"
) )
if not self._is_optional_str(return_ann):
raise TypeError(
f"logfmt class {cls.__name__}.resolve must return str | None, "
f"got {return_ann!r}"
)
return _LogFmtClassSpec(cls=cls, inject_params=inject_params, path=path) return _LogFmtClassSpec(cls=cls, inject_params=inject_params, path=path)
@@ -439,7 +568,7 @@ class CallbackRegistry:
) -> dict[str, Any]: ) -> dict[str, Any]:
kwargs: dict[str, Any] = {} kwargs: dict[str, Any] = {}
for name, ann in params: for name, ann in params:
value = self._resolve_annotation(ann, ctx) value = self._resolve_annotation(name, ann, ctx)
if value is _UNRESOLVED: if value is _UNRESOLVED:
raise RuntimeError(f"no value available for annotation {ann!r}") raise RuntimeError(f"no value available for annotation {ann!r}")
kwargs[name] = value kwargs[name] = value
@@ -447,41 +576,54 @@ class CallbackRegistry:
def _is_allowed(self, kind: str, ann: Any) -> bool: def _is_allowed(self, kind: str, ann: Any) -> bool:
bare = self._unwrap_optional(ann) bare = self._unwrap_optional(ann)
if self._matches_state_annotation(bare, "pre"): if _state_tag(bare) is not None:
return kind == "logfmt"
if self._matches_state_annotation(bare, "post"):
return kind == "logfmt" return kind == "logfmt"
if bare is DatabaseError: if bare is DatabaseError:
return kind == "fatal_error" return kind == "fatal_error"
if bare is MigrationReport:
return kind == "logmigr"
if self._data_type is not None and bare is self._data_type: if self._data_type is not None and bare is self._data_type:
return kind == "bootstrap" return kind in {"bootstrap", "validate"}
if self._kanta_class is not None and bare is self._kanta_class: if self._kanta_class is not None and bare is self._kanta_class:
return kind in {"bootstrap", "fatal_error", "logfmt"} return kind in {
"bootstrap",
"fatal_error",
"logfmt",
"logmigr",
"validate",
}
return False return False
def _allowed_message(self, kind: str) -> str: def _allowed_message(self, kind: str) -> str:
parts: list[str] = [] parts: list[str] = []
if kind == "bootstrap": if kind in {"bootstrap", "validate"}:
if self._data_type is not None: if self._data_type is not None:
parts.append(self._data_type.__name__) parts.append(self._data_type.__name__)
if kind in {"bootstrap", "fatal_error", "logfmt"}: if kind in {"bootstrap", "fatal_error", "logfmt", "logmigr", "validate"}:
if self._kanta_class is not None: if self._kanta_class is not None:
parts.append(self._kanta_class.__name__) parts.append(self._kanta_class.__name__)
if kind == "fatal_error": if kind == "fatal_error":
parts.append("DatabaseError") parts.append("DatabaseError")
if kind == "logmigr":
parts.append("MigrationReport")
if kind == "logfmt": if kind == "logfmt":
parts.append("Annotated[dict, 'pre']") parts.append("prev: dict")
parts.append("Annotated[dict, 'post']") parts.append("state: dict")
return ", ".join(parts) if parts else "none" return ", ".join(parts) if parts else "none"
def _resolve_annotation(self, ann: Any, ctx: InjectionContext) -> Any: def _resolve_annotation(self, name: str, ann: Any, ctx: InjectionContext) -> Any:
bare = self._unwrap_optional(ann) bare = self._unwrap_optional(ann)
if self._matches_state_annotation(bare, "pre"): # An explicit tag takes precedence over the parameter name.
return ctx.previous_state tag = _state_tag(bare)
if self._matches_state_annotation(bare, "post"): if tag is not None:
return ctx.current_state return getattr(ctx, _STATE_KINDS[tag])
key = _state_key(name)
if key is not None:
return getattr(ctx, _STATE_KINDS[key])
if bare is DatabaseError: if bare is DatabaseError:
return ctx.error return ctx.error
if bare is MigrationReport:
return ctx.report
if self._data_type is not None and bare is self._data_type: if self._data_type is not None and bare is self._data_type:
return ctx.data return ctx.data
if self._kanta_class is not None and bare is self._kanta_class: if self._kanta_class is not None and bare is self._kanta_class:
@@ -503,20 +645,10 @@ class CallbackRegistry:
) from exc ) from exc
return raw_ann return raw_ann
@staticmethod
def _matches_state_annotation(ann: Any, marker: str) -> bool:
origin = get_origin(ann)
if origin is not Annotated:
return False
args = get_args(ann)
if not args:
return False
return args[0] is dict and marker in args[1:]
@staticmethod @staticmethod
def _unwrap_optional(ann: Any) -> Any: def _unwrap_optional(ann: Any) -> Any:
origin = get_origin(ann) origin = get_origin(ann)
if origin is not Union: if origin not in (Union, types.UnionType):
return ann return ann
args = [arg for arg in get_args(ann) if arg is not type(None)] args = [arg for arg in get_args(ann) if arg is not type(None)]
return args[0] if len(args) == 1 else ann return args[0] if len(args) == 1 else ann
@@ -524,7 +656,7 @@ class CallbackRegistry:
@staticmethod @staticmethod
def _is_optional_str(ann: Any) -> bool: def _is_optional_str(ann: Any) -> bool:
origin = get_origin(ann) origin = get_origin(ann)
if origin is not Union: if origin not in (Union, types.UnionType):
return ann is str return ann is str
args = get_args(ann) args = get_args(ann)
return type(None) in args and any(arg is str for arg in args) return type(None) in args and any(arg is str for arg in args)
+51 -48
View File
@@ -1,63 +1,66 @@
"""Diff computation and replay utilities.""" """Diff computation and replay utilities.
import jsondiff Diff format: JSON-serializable dicts where ``$delete`` is the only command
our producer emits; added keys and changed values (scalars, lists, type
changes — lists always wholesale) are plain assignments. A dict value
assigned over a non-dict needs no ``$replace``: the consumer can see from
the old value whether to patch (old is a dict) or replace. User keys
starting with ``$`` are escaped by prepending another ``$``
(``$foo`` -> ``$$foo``); values are stored verbatim.
The consumer additionally stays compatible with jsondiff's marshaled
syntax, so it can replay diffs produced by jsondiff itself: ``$replace``,
positional ``$insert``/``$delete`` and per-index nested diffs on lists,
and jsondiff's escaping of ``$``-prefixed values.
"""
from kanta.structs import ChangeRecord from kanta.structs import ChangeRecord
from kanta.serialization.base import ReplayResult, replay from kanta.serialization.base import ReplayResult, apply_diff, replay
from kanta.serialization.framing import LineFramer from kanta.serialization.framing import LineFramer
from kanta.serialization.json import JsonSerializer from kanta.serialization.json import JsonSerializer
_UNCHANGED = object()
def compute_diff(previous: dict, current: dict) -> dict | None:
"""Compute a jsondiff patch between two dicts. def _escape_key(key: str) -> str:
"""Escape a user key for use as a diff key (``$foo`` -> ``$$foo``)."""
if isinstance(key, str) and key.startswith("$"):
return "$" + key
return key
def _diff(previous, current):
"""Compute a raw diff, or _UNCHANGED if there is no difference."""
if isinstance(previous, dict) and isinstance(current, dict):
result = {}
deleted = [_escape_key(k) for k in previous if k not in current]
if deleted:
result["$delete"] = deleted
for key, new_value in current.items():
if key not in previous:
result[_escape_key(key)] = new_value
else:
sub = _diff(previous[key], new_value)
if sub is not _UNCHANGED:
result[_escape_key(key)] = sub
return result if result else _UNCHANGED
if previous == current:
return _UNCHANGED
return current
def diff(previous: dict, current: dict) -> dict | None:
"""Compute a marshaled diff between two state dicts.
Returns None if there is no difference. Returns None if there is no difference.
""" """
return jsondiff.diff(previous, current, marshal=True) or None result = _diff(previous, current)
return result if result is not _UNCHANGED else None
def _apply_diff(state: dict, diff: dict) -> dict: def patch(state: dict, diff: dict) -> dict:
"""Apply a jsondiff patch manually, handling ``$replace`` and ``$delete``. """Apply a marshaled diff to a state dict."""
return apply_diff(state, diff)
jsondiff.patch does not handle nested ``$replace`` commands when the
parent key is missing from the state. This function recursively applies
diffs, treating ``$replace`` as full replacement and ``$delete`` as
key removal.
"""
if not isinstance(diff, dict):
return diff
result = dict(state) if isinstance(state, dict) else state
if not isinstance(result, dict):
result = {}
for key, value in diff.items():
if key == "$replace":
return value
elif key == "$delete":
if isinstance(value, list):
for k in value:
result.pop(k, None)
else:
result.pop(value, None)
elif isinstance(value, dict):
old = result.get(key, {})
if not isinstance(old, dict):
old = {}
result[key] = _apply_diff(old, value)
else:
result[key] = value
return result
def patch_state(state: dict, diff: dict) -> dict:
"""Apply a jsondiff patch to a state dict.
The diff was produced with ``marshal=True`` (string keys like
``"$replace"`` and ``"$delete"``) and decoded from JSON.
"""
return _apply_diff(state, diff)
# Backward-compatible JSONL replay using the default serializer. # Backward-compatible JSONL replay using the default serializer.
+93 -27
View File
@@ -16,7 +16,7 @@ from pathlib import Path
from kanta.exceptions import FileLockError from kanta.exceptions import FileLockError
_logger = logging.getLogger(__name__) _logger = logging.getLogger("kanta")
def _fatal(msg: str, *, db_path: Path | None = None) -> None: def _fatal(msg: str, *, db_path: Path | None = None) -> None:
@@ -34,6 +34,7 @@ if sys.platform == "win32":
_GENERIC_READ = 0x80000000 _GENERIC_READ = 0x80000000
_GENERIC_WRITE = 0x40000000 _GENERIC_WRITE = 0x40000000
_FILE_SHARE_READ = 0x00000001 _FILE_SHARE_READ = 0x00000001
_FILE_SHARE_WRITE = 0x00000002
_OPEN_EXISTING = 3 _OPEN_EXISTING = 3
_OPEN_ALWAYS = 4 _OPEN_ALWAYS = 4
_FILE_ATTRIBUTE_NORMAL = 0x80 _FILE_ATTRIBUTE_NORMAL = 0x80
@@ -82,6 +83,10 @@ if sys.platform == "win32":
] ]
_kernel32.CloseHandle.restype = wintypes.BOOL _kernel32.CloseHandle.restype = wintypes.BOOL
_kernel32.CloseHandle.argtypes = [wintypes.HANDLE] _kernel32.CloseHandle.argtypes = [wintypes.HANDLE]
_kernel32.SetEndOfFile.restype = wintypes.BOOL
_kernel32.SetEndOfFile.argtypes = [wintypes.HANDLE]
_kernel32.FlushFileBuffers.restype = wintypes.BOOL
_kernel32.FlushFileBuffers.argtypes = [wintypes.HANDLE]
def _is_invalid_handle(handle) -> bool: def _is_invalid_handle(handle) -> bool:
return ctypes.c_void_p(handle).value == ctypes.c_void_p(-1).value return ctypes.c_void_p(handle).value == ctypes.c_void_p(-1).value
@@ -91,15 +96,16 @@ else:
class LockedFile: class LockedFile:
"""A file opened with an exclusive write lock. """A file opened for read+write with an optional exclusive lock.
Usage:: Usage::
f = LockedFile() f = LockedFile()
f.open(path) # open + lock (read+write) f.open(path) # open + lock (read+write)
content = f.read() # read entire content f.open(path, readonly=True) # open read-only without locking
f.write(data) # append data (seeks to end first) content = f.read() # read entire content
f.close() # release lock + close fd f.write(data) # append data (seeks to end first)
f.close() # release lock + close fd
Unix: fcntl.flock (advisory) — read-only callers that don't flock are unaffected. Unix: fcntl.flock (advisory) — read-only callers that don't flock are unaffected.
Windows: CreateFileW with FILE_SHARE_READ — OS blocks other writers. Windows: CreateFileW with FILE_SHARE_READ — OS blocks other writers.
@@ -108,12 +114,13 @@ class LockedFile:
def __init__(self) -> None: def __init__(self) -> None:
self._fd: int | None = None # Unix fd or Windows HANDLE self._fd: int | None = None # Unix fd or Windows HANDLE
def open(self, path: Path, *, create: bool = False) -> None: def open(self, path: Path, *, create: bool = False, readonly: bool = False) -> None:
"""Open *path* for read+write with an exclusive lock. """Open *path* and optionally acquire an exclusive lock.
Args: Args:
path: File to open and lock. path: File to open and lock.
create: If True, create the file if it doesn't exist (bootstrap). create: If True, create the file if it doesn't exist (bootstrap).
readonly: If True, open read-only without acquiring a lock.
Raises: Raises:
FileLockError: If the file is locked by another process or not found. FileLockError: If the file is locked by another process or not found.
@@ -122,16 +129,18 @@ class LockedFile:
return # Already open (idempotent) return # Already open (idempotent)
if sys.platform == "win32": if sys.platform == "win32":
self._open_win32(path, create) self._open_win32(path, create, readonly)
else: else:
self._open_unix(path, create) self._open_unix(path, create, readonly)
def open_and_read(self, path: Path, create: bool = False) -> bytes: def open_and_read(
"""Open *path* with exclusive lock and read all content. self, path: Path, create: bool = False, readonly: bool = False
) -> bytes:
"""Open *path* and read all content.
Combined operation for efficient use with asyncio.to_thread(). Combined operation for efficient use with asyncio.to_thread().
""" """
self.open(path, create=create) self.open(path, create=create, readonly=readonly)
return self.read() return self.read()
def read(self) -> bytes: def read(self) -> bytes:
@@ -172,6 +181,22 @@ class LockedFile:
os.lseek(self._fd, current, os.SEEK_SET) os.lseek(self._fd, current, os.SEEK_SET)
return end return end
def replace_content(self, data: bytes) -> None:
"""Atomically-ish rewrite the file's content in place, lock retained.
Seeks to the start, truncates, writes *data* and fsyncs, all on the
already-locked descriptor. The path is never unlinked or renamed, so
no other process can observe a missing file or acquire its own lock.
Used by database rotation.
"""
if self._fd is None:
raise RuntimeError("LockedFile.replace_content() called on a closed file")
if sys.platform == "win32":
self._replace_content_win32(data)
else:
self._replace_content_unix(data)
def close(self) -> None: def close(self) -> None:
"""Release the lock and close the file.""" """Release the lock and close the file."""
if self._fd is None: if self._fd is None:
@@ -188,20 +213,24 @@ class LockedFile:
# -- Unix ---------------------------------------------------------------- # -- Unix ----------------------------------------------------------------
def _open_unix(self, path: Path, create: bool) -> None: def _open_unix(self, path: Path, create: bool, readonly: bool) -> None:
flags = os.O_RDWR | (os.O_CREAT if create else 0) if readonly:
flags = os.O_RDONLY
else:
flags = os.O_RDWR | (os.O_CREAT if create else 0)
try: try:
fd = os.open(path, flags, 0o666) fd = os.open(path, flags, 0o666)
except FileNotFoundError: except FileNotFoundError:
_fatal(f"Database file not found: {path.resolve()}", db_path=path) _fatal(f"Database file not found: {path.resolve()}", db_path=path)
try: if not readonly:
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) try:
except OSError: fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
os.close(fd) except OSError:
_fatal( os.close(fd)
f"{path.resolve()}: database already locked by another instance", _fatal(
db_path=path, f"{path.resolve()}: database already locked by another instance",
) db_path=path,
)
self._fd = fd self._fd = fd
def _read_unix(self) -> bytes: def _read_unix(self) -> bytes:
@@ -218,14 +247,30 @@ class LockedFile:
os.lseek(self._fd, 0, os.SEEK_END) os.lseek(self._fd, 0, os.SEEK_END)
os.write(self._fd, data) os.write(self._fd, data)
def _replace_content_unix(self, data: bytes) -> None:
os.lseek(self._fd, 0, os.SEEK_SET)
os.ftruncate(self._fd, 0)
view = memoryview(data)
while view:
written = os.write(self._fd, view)
view = view[written:]
os.fdatasync(self._fd)
# -- Windows ------------------------------------------------------------- # -- Windows -------------------------------------------------------------
def _open_win32(self, path: Path, create: bool) -> None: def _open_win32(self, path: Path, create: bool, readonly: bool) -> None:
disposition = _OPEN_ALWAYS if create else _OPEN_EXISTING if readonly:
disposition = _OPEN_EXISTING
access = _GENERIC_READ
share = _FILE_SHARE_READ | _FILE_SHARE_WRITE
else:
disposition = _OPEN_ALWAYS if create else _OPEN_EXISTING
access = _GENERIC_READ | _GENERIC_WRITE
share = _FILE_SHARE_READ
handle = _kernel32.CreateFileW( handle = _kernel32.CreateFileW(
str(path), str(path),
_GENERIC_READ | _GENERIC_WRITE, access,
_FILE_SHARE_READ, share,
None, None,
disposition, disposition,
_FILE_ATTRIBUTE_NORMAL, _FILE_ATTRIBUTE_NORMAL,
@@ -272,3 +317,24 @@ class LockedFile:
) )
if not ok: if not ok:
raise OSError(f"WriteFile failed: Windows error {ctypes.get_last_error()}") raise OSError(f"WriteFile failed: Windows error {ctypes.get_last_error()}")
def _replace_content_win32(self, data: bytes) -> None:
_kernel32.SetFilePointer(self._fd, 0, None, _FILE_BEGIN)
written = wintypes.DWORD()
ok = _kernel32.WriteFile(
self._fd,
data,
len(data),
ctypes.byref(written),
None,
)
if not ok:
raise OSError(f"WriteFile failed: Windows error {ctypes.get_last_error()}")
if not _kernel32.SetEndOfFile(self._fd):
raise OSError(
f"SetEndOfFile failed: Windows error {ctypes.get_last_error()}"
)
if not _kernel32.FlushFileBuffers(self._fd):
raise OSError(
f"FlushFileBuffers failed: Windows error {ctypes.get_last_error()}"
)
+463
View File
@@ -0,0 +1,463 @@
"""Structural grep matching and match highlighting for change records.
Backs the ``--grep`` option of the ``python -m kanta`` CLI. Patterns
match against the structure of a transaction — its action, user, and the
dotted paths and values of its diff — never against rendered output
text. Matching is case-insensitive.
Path terms match element-wise: each dotted element of the term must match
a whole path element (``users`` matches ``users`` anywhere in the path
but not ``foousers``), unless the element uses shell wildcards
(``us*``). The term's elements match as a contiguous sequence, so
``users.*.email`` matches the path ``users.alice.email``. Only the
matched elements are highlighted.
Value terms match strings by substring and all other values (booleans,
numbers, null) only in full: ``true`` matches a boolean but ``tru`` does
not. A term with wildcards matches the whole value text. Matched
substrings — or the whole scalar — are highlighted. When a logfmt
formatter (``-k``) prettifies a value or the user, both the raw and the
prettified form are matched.
A matched record carries a :class:`GrepHighlighter`, which the
:kanta.logging formatters consult to wrap exactly the matched regions
with a yellow background.
"""
from __future__ import annotations
import dataclasses
import fnmatch
import json
from collections.abc import Iterator
from typing import Any
from kanta.logging import _USER_PATH, _collect_changes, _get_nested
from kanta.serialization.base import unmarshal
from kanta.structs import ChangeRecord
from kanta.tty import ANSI_RE, ESC, strip_ansi
_GLOB_CHARS = frozenset("*?[")
_MARK_BG = f"{ESC}48;5;220m" # yellow background (xterm256 #ffd700) for matches
_UNMARK_BG = f"{ESC}49m" # back to the default background, foreground untouched
def mark_spans(styled: str, spans: list[tuple[int, int]]) -> str:
"""Wrap the given visible-text spans of a styled string with the mark color.
``spans`` are ``(start, end)`` offsets into the visible text of
*styled*; ANSI sequences are not counted. Overlapping and adjacent
spans are merged first, so overlapping matches from different patterns
produce one continuous highlight. The set/clear codes are inserted at
the mapped positions in *styled*; only the background attribute is
touched, leaving foreground colors intact.
"""
spans = _merge_spans(spans)
if not spans:
return styled
out: list[str] = []
plain_pos = 0
prev_end = 0
for match in ANSI_RE.finditer(styled):
out.append(_wrap_run(styled[prev_end : match.start()], plain_pos, spans))
plain_pos += match.start() - prev_end
out.append(match.group(0))
prev_end = match.end()
out.append(_wrap_run(styled[prev_end:], plain_pos, spans))
return "".join(out)
def _merge_spans(spans: list[tuple[int, int]]) -> list[tuple[int, int]]:
"""Return *spans* sorted, with overlapping and adjacent spans merged."""
merged: list[list[int]] = []
for start, end in sorted(spans):
if start >= end:
continue
if merged and start <= merged[-1][1]:
merged[-1][1] = max(end, merged[-1][1])
else:
merged.append([start, end])
return [(start, end) for start, end in merged]
def _wrap_run(run: str, plain_start: int, spans: list[tuple[int, int]]) -> str:
"""Wrap the intersections of *spans* with one escape-free text run."""
if not run:
return run
out: list[str] = []
pos = 0
for start, end in spans:
s = max(start - plain_start, 0)
e = min(end - plain_start, len(run))
if s >= e or e <= pos:
continue
out.append(run[pos:s])
out.append(f"{_MARK_BG}{run[s:e]}{_UNMARK_BG}")
pos = e
out.append(run[pos:])
return "".join(out)
def _find_spans(text: str, needle: str) -> list[tuple[int, int]]:
"""Return visible-text spans of every case-insensitive occurrence of *needle*."""
if not needle:
return []
haystack = strip_ansi(text).lower()
needle = needle.lower()
spans = []
pos = 0
while (found := haystack.find(needle, pos)) >= 0:
end = found + len(needle)
spans.append((found, end))
pos = end
return spans
@dataclasses.dataclass(frozen=True)
class GrepPattern:
"""One parsed ``--grep`` pattern.
The bare form (``term`` set) matches the action, the user, or any
dotted path or value in the diff. The ``path=value`` form
(``path_term`` and ``value_term`` set) requires both sides to match
within the same change line; either side may be left empty to match
values only (``=value``) or paths only (``path=``).
"""
raw: str
term: str | None = None
path_term: str | None = None
value_term: str | None = None
@classmethod
def parse(cls, raw: str) -> GrepPattern:
"""Parse a pattern, splitting the ``path=value`` form on the first ``=``."""
if "=" in raw:
path_term, value_term = raw.split("=", 1)
return cls(raw, path_term=path_term, value_term=value_term)
return cls(raw, term=raw)
@dataclasses.dataclass(frozen=True)
class _ValueMark:
"""A value-side match.
``needle`` is the text to locate in the displayed value ("" = a match
that marks nothing, e.g. from an empty term). ``whole`` marks the
entire displayed value: used when the raw value matched but a logfmt
formatter displays something else, so no needle can be located.
"""
needle: str = ""
whole: bool = False
@dataclasses.dataclass
class _Entry:
"""One matchable ``(path, value)`` line of a record's flattened diff.
``anchor`` is set for deleted content: the deleted path whose line is
displayed for this entry (the entry itself may sit below it).
"""
path: list[str]
raw: Any
text: str
anchor: list[str] | None = None
def _element_matches(pattern: str, element: str) -> bool:
"""Match one path element: in full, or as a glob when it uses wildcards."""
pattern = pattern.casefold()
element = element.casefold()
if any(char in pattern for char in _GLOB_CHARS):
return fnmatch.fnmatchcase(element, pattern)
return element == pattern
def _match_path(term: str, path: list[str]) -> frozenset[int] | None:
"""Match a dotted term against *path* as a contiguous element sequence.
Returns the indices of the matched elements, or ``None``. An empty
term matches anything and marks no elements.
"""
if not term:
return frozenset()
patterns = term.split(".")
for start in range(len(path) - len(patterns) + 1):
if all(
_element_matches(pattern, path[start + i])
for i, pattern in enumerate(patterns)
):
return frozenset(range(start, start + len(patterns)))
return None
def _match_value(term: str, raw: Any, text: str) -> _ValueMark | None:
"""Match a term against a value.
String values match by substring; containers do not match (their
leaves are matched individually); all other values match only in
full. A term with wildcards matches the whole value text.
"""
if not term:
return _ValueMark()
needle = term.casefold()
haystack = text.casefold()
if any(char in needle for char in _GLOB_CHARS):
return _ValueMark(text) if fnmatch.fnmatchcase(haystack, needle) else None
if isinstance(raw, str):
return _ValueMark(term) if needle in haystack else None
if isinstance(raw, (dict, list)):
return None
return _ValueMark(text) if needle == haystack else None
def _dual_mark(term: str, raw: Any, text: str, pretty: str | None) -> _ValueMark | None:
"""Match a term against both the raw and the prettified form of a value.
*pretty* is the logfmt-resolved display text, which is what the log
shows when set. A match on the displayed form locates its needle
there; a match on the raw form alone marks the whole displayed value.
"""
raw_mark = _match_value(term, raw, text)
if pretty is None:
return raw_mark
pretty_mark = _match_value(term, pretty, pretty)
if pretty_mark is not None:
return pretty_mark
if raw_mark is not None and (raw_mark.needle or raw_mark.whole):
return _ValueMark(whole=True)
return raw_mark
def _value_text(value: Any) -> str:
"""Render a value as matchable text, following the display conventions."""
if value is None:
return "null"
if isinstance(value, bool):
return "true" if value else "false"
if isinstance(value, (int, float)):
return str(value)
if isinstance(value, str):
return value
if isinstance(value, (dict, list)):
try:
return json.dumps(value, default=str)
except (TypeError, ValueError):
pass
return str(value)
def _leaf_entries(
path: list[str], value: Any, anchor: list[str] | None
) -> Iterator[_Entry]:
"""Yield an entry for every value under *value*.
Added, replaced or deleted containers are a single change line in the
log but hold many values; descending into them lets patterns match
their content. List elements are addressed by index (``users.0``).
"""
items: Iterator[tuple[str, Any]]
if isinstance(value, dict):
items = ((str(key), item) for key, item in value.items())
elif isinstance(value, list):
items = ((str(index), item) for index, item in enumerate(value))
else:
return
for key, item in items:
item_path = [*path, key]
yield _Entry(item_path, item, _value_text(item), anchor)
yield from _leaf_entries(item_path, item, anchor)
def _record_entries(record: ChangeRecord, previous: dict | None) -> list[_Entry]:
"""Flatten a record's diff into matchable entries.
Uses the same traversal as the change-log rendering, so paths match
what the log shows; deleted paths carry their previous value.
"""
changes: list[tuple[str, list[str], Any]] = []
_collect_changes(unmarshal(record.diff), [], changes, previous)
entries: list[_Entry] = []
for change_type, path, value in changes:
anchor = None
if change_type == "delete":
value = _get_nested(previous, path)
anchor = path
entries.append(_Entry(path, value, _value_text(value), anchor))
entries.extend(_leaf_entries(path, value, anchor))
return entries
class GrepHighlighter:
"""The matched regions of one record, wrapping rendered text on demand.
Implements the highlighter hook of the :mod:`kanta.logging`
formatters: ``path`` for path elements, ``value`` for values,
``meta`` for header fields and ``delete`` for deletion markers.
All wrapping goes through :func:`mark_spans`, so overlapping matches
merge into one highlight.
"""
def __init__(self) -> None:
self._lit_paths: set[str] = set()
self._lit_deletes: set[str] = set()
self._value_marks: dict[str, list[_ValueMark]] = {}
self._meta_marks: dict[str, list[_ValueMark]] = {}
def _light_elements(self, path: list[str], indices) -> None:
for i in indices:
self._lit_paths.add(".".join(path[: i + 1]))
def add(
self,
entry: _Entry,
elements: frozenset[int] | None,
vmark: _ValueMark | None,
) -> None:
"""Record one entry's match: lit element indices and/or a value mark."""
marked = vmark is not None and (vmark.needle or vmark.whole)
if entry.anchor is None:
if elements:
self._light_elements(entry.path, elements)
if marked:
key = ".".join(entry.path)
self._value_marks.setdefault(key, []).append(vmark)
return
# Deleted content: only the anchor path line is displayed. Light
# the genuinely matched elements within it; a match on the removed
# value or below the anchor marks the deletion marker (✗) instead.
if elements:
shown = {i for i in elements if i < len(entry.anchor)}
if shown:
self._light_elements(entry.path, shown)
if len(shown) != len(elements):
self._lit_deletes.add(".".join(entry.anchor))
if marked:
self._lit_deletes.add(".".join(entry.anchor))
def add_meta(self, field: str, mark: _ValueMark) -> None:
"""Record a header match on ``field`` (``"action"`` or ``"user"``)."""
if mark.needle or mark.whole:
self._meta_marks.setdefault(field, []).append(mark)
def path(self, text: str, path: str) -> str:
"""Wrap a rendered path element when its element matched."""
if text and path in self._lit_paths:
return mark_spans(text, [(0, len(strip_ansi(text)))])
return text
def delete(self, text: str, path: str) -> str:
"""Wrap the deletion marker when the removed content matched."""
if text and path in self._lit_deletes:
return mark_spans(text, [(0, len(strip_ansi(text)))])
return text
@staticmethod
def _apply_marks(text: str, marks: list[_ValueMark]) -> str:
if not text or not marks:
return text
spans: list[tuple[int, int]] = []
for mark in marks:
if mark.whole:
spans.append((0, len(strip_ansi(text))))
else:
spans.extend(_find_spans(text, mark.needle))
return mark_spans(text, spans)
def value(self, text: str, path: str) -> str:
"""Wrap the matched regions of a rendered value."""
return self._apply_marks(text, self._value_marks.get(path, []))
def meta(self, text: str, field: str) -> str:
"""Wrap the matched regions of a rendered header field."""
return self._apply_marks(text, self._meta_marks.get(field, []))
def _match_entry(
pattern: GrepPattern, entry: _Entry, logfmt: Any
) -> tuple[frozenset[int] | None, _ValueMark | None] | None:
"""Match one pattern against one entry, returning its match marks.
Returns ``None`` when the entry does not match. Otherwise returns the
matched path-element indices and/or the value mark, ready for
:meth:`GrepHighlighter.add`.
"""
pretty = None
if logfmt is not None:
pretty = logfmt(entry.raw, ".".join(entry.path))
if pattern.term is not None:
elements = _match_path(pattern.term, entry.path)
vmark = _dual_mark(pattern.term, entry.raw, entry.text, pretty)
if elements is None and vmark is None:
return None
else:
elements = _match_path(pattern.path_term or "", entry.path)
vmark = _dual_mark(pattern.value_term or "", entry.raw, entry.text, pretty)
if elements is None or vmark is None:
return None
return elements, vmark
def matches_snapshot(
state: dict, patterns: list[GrepPattern], logfmt: Any = None
) -> bool:
"""Match *patterns* against a snapshot's full state.
The state is flattened into the same ``(path, value)`` entries change
records are matched against, so path and value matching semantics are
identical; a snapshot simply has no action or user to match. Returns
whether every pattern matched somewhere in the state.
"""
entries = list(_leaf_entries([], unmarshal(state), None))
for pattern in patterns:
if not any(_match_entry(pattern, entry, logfmt) for entry in entries):
return False
return True
def evaluate(
record: ChangeRecord,
previous: dict | None,
patterns: list[GrepPattern],
logfmt: Any = None,
) -> GrepHighlighter | None:
"""Match *patterns* against a record, returning its matched regions.
Returns ``None`` when any pattern matches nowhere in the transaction.
Otherwise every pattern contributed its matches — action, user, or
change lines — to the returned highlighter; different patterns may
match different lines of the same record.
``logfmt`` is the optional composed logfmt callable; when given, both
the raw and the prettified form of each value (and of the user) are
matched.
"""
entries = _record_entries(record, previous)
highlighter = GrepHighlighter()
for pattern in patterns:
matched = False
if pattern.term is not None:
mark = _match_value(pattern.term, record.a, record.a)
if mark is not None:
highlighter.add_meta("action", mark)
matched = True
if record.u:
pretty_user = (
logfmt(record.u, _USER_PATH) if logfmt is not None else None
)
mark = _dual_mark(pattern.term, record.u, record.u, pretty_user)
if mark is not None:
highlighter.add_meta("user", mark)
matched = True
for entry in entries:
match = _match_entry(pattern, entry, logfmt)
if match is None:
continue
matched = True
highlighter.add(entry, *match)
if not matched:
return None
return highlighter
+146 -7
View File
@@ -1,9 +1,10 @@
"""Kanta DB main public API""" """Kanta DB main public API"""
from __future__ import annotations from __future__ import annotations
from datetime import datetime import logging
from datetime import datetime, timedelta
from pathlib import Path from pathlib import Path
from types import ModuleType from types import ModuleType, SimpleNamespace
from typing import Any, Generic, TypeVar from typing import Any, Generic, TypeVar
from kanta.kantaimpl import KantaImpl from kanta.kantaimpl import KantaImpl
@@ -50,9 +51,9 @@ class Kanta(Generic[T]):
*, *,
type: type[T] | None = None, type: type[T] | None = None,
migrations: ModuleType | str | None = None, migrations: ModuleType | str | None = None,
migration_ctx: Any | None = None,
serializer: Serializer | None = None, serializer: Serializer | None = None,
flush_interval: float = 0.1, flush_interval: float = 0.1,
retention: timedelta | int | None = None,
): ):
"""Initialize a Kanta persistence instance. """Initialize a Kanta persistence instance.
@@ -61,9 +62,15 @@ class Kanta(Generic[T]):
data: Caller-owned root msgspec.Struct state instance. data: Caller-owned root msgspec.Struct state instance.
type: Optional explicit root type. Defaults to ``type(data)``. type: Optional explicit root type. Defaults to ``type(data)``.
migrations: Optional migrations module object or import path. migrations: Optional migrations module object or import path.
migration_ctx: Optional context object passed to migration functions.
flush_interval: Background flush interval in seconds. flush_interval: Background flush interval in seconds.
serializer: Optional serializer implementation. serializer: Optional serializer implementation.
retention: Optional history retention window, either a
:class:`~datetime.timedelta` or a plain number of days. When
set, opening the database rotates it: history older than
``now - retention`` is moved to a ``{stem}@{timestamp}.kantadb``
sibling file and the main file is rewritten with a fresh
snapshot plus the retained records (see ``docs/rotation.md``).
``None`` (default) disables rotation.
Raises: Raises:
ImportError: If ``migrations`` is a string path that cannot be imported. ImportError: If ``migrations`` is a string path that cannot be imported.
@@ -78,8 +85,8 @@ class Kanta(Generic[T]):
data=data, data=data,
type=data_type, type=data_type,
migrations=migrations, migrations=migrations,
migration_ctx=migration_ctx,
flush_interval=flush_interval, flush_interval=flush_interval,
retention=retention,
kanta=self, kanta=self,
) )
@@ -127,6 +134,18 @@ class Kanta(Generic[T]):
""" """
return self._impl.filename return self._impl.filename
@property
def ctx(self) -> SimpleNamespace:
"""User-writable context namespace.
Migration functions receive the ``Kanta`` instance and can read or
mutate ``kanta.ctx`` during migrations. Applications can also store
arbitrary data here (e.g. a connection id); since
:class:`kanta.logging.LogEvent` carries the Kanta instance, logemit
callbacks can read it as ``event.kanta.ctx``.
"""
return self._impl.ctx
@property @property
def mtime(self) -> datetime | None: def mtime(self) -> datetime | None:
"""Last modification time carried forward from change records. """Last modification time carried forward from change records.
@@ -138,7 +157,13 @@ class Kanta(Generic[T]):
""" """
return self._impl.mtime return self._impl.mtime
async def open(self, *, create: bool = True) -> None: async def open(
self,
*,
create: bool = True,
readonly: bool = False,
log: bool | logging.Logger = True,
) -> None:
"""Open the database file and start background persistence. """Open the database file and start background persistence.
This loads existing records, applies configured migrations, and starts This loads existing records, applies configured migrations, and starts
@@ -147,6 +172,16 @@ class Kanta(Generic[T]):
Args: Args:
create: Whether to create the database file when missing. create: Whether to create the database file when missing.
If False, opening fails when the file does not exist or is empty. If False, opening fails when the file does not exist or is empty.
readonly: If True, open the database read-only. No lock is acquired,
no background flush task is started, and transactions are
rejected. The file is not created if missing.
log: Controls bootstrap and migration logging. ``True`` (default)
uses the ``kanta.bootstrap`` logger for bootstrap records and
the ``kanta.migration`` logger for migration output. ``False``
suppresses the default bootstrap and migration logs. A
:class:`~logging.Logger` instance writes default output to that
logger instead. Custom ``@kanta.logmigr`` callbacks run
regardless of this setting.
Calling ``open`` more than once on the same instance is not allowed. Calling ``open`` more than once on the same instance is not allowed.
@@ -154,7 +189,7 @@ class Kanta(Generic[T]):
kanta.exceptions.DatabaseError: If replay or decoding fails. kanta.exceptions.DatabaseError: If replay or decoding fails.
kanta.exceptions.DataIntegrityError: If the instance is already open. kanta.exceptions.DataIntegrityError: If the instance is already open.
""" """
await self._impl.open(create=create) await self._impl.open(create=create, readonly=readonly, log=log)
async def __aenter__(self) -> Kanta[T]: async def __aenter__(self) -> Kanta[T]:
"""Enter async context manager and open the database. """Enter async context manager and open the database.
@@ -214,6 +249,27 @@ class Kanta(Generic[T]):
return _register return _register
return _register(fn) return _register(fn)
def validate(self, fn):
"""Register a data validation callback.
Used as ``@kanta.validate``. The callback receives the live data
object (and optionally the ``Kanta`` instance) and must raise an
exception when the data is inconsistent. Validators run after replay
during :meth:`open` (after msgspec decoding and migrations) and after
each transaction, before the change is committed to history. Multiple
validators run in registration order until the first failure.
Validators must be synchronous and must not modify the data — they
only fail. A failure inside a transaction rolls the transaction back;
a failure during open aborts the open.
"""
def _register(callback):
self._impl.add_validate(callback)
return callback
return _register(fn)
def fatal_error(self, fn=None): def fatal_error(self, fn=None):
"""Register fatal error handler callback. """Register fatal error handler callback.
@@ -230,6 +286,45 @@ class Kanta(Generic[T]):
return _register return _register
return _register(fn) return _register(fn)
def clock(self, fn=None):
"""Register a clock callback replacing the default UTC clock.
Can be used as ``@kanta.clock``. The callback takes no arguments and
must return a :class:`~datetime.datetime`; its value is used for all
record timestamps (``ts``, and ``m`` when ``mtime`` is ``True``) and
snapshot timestamps. The clock is only read when a timestamp is
actually produced, so read-count-dependent clocks (e.g. advancing on
every read) stay deterministic. Register before :meth:`open` so that
bootstrap and migration records use the custom clock as well. This is
mainly useful for tests and reproducible demos.
"""
def _register(callback):
self._impl.add_clock(callback)
return callback
if fn is None:
return _register
return _register(fn)
def logmigr(self, fn=None):
"""Register a migration logging callback.
Can be used as ``@kanta.logmigr``.
The callback receives a :class:`kanta.migrations.MigrationReport` and
may be sync or async. If registered, it replaces the default migration
logger output; the application is responsible for emitting any log
messages.
"""
def _register(callback):
self._impl.add_logmigr(callback)
return callback
if fn is None:
return _register
return _register(fn)
def logfmt(self, fn=None, *, path: str | None = None): def logfmt(self, fn=None, *, path: str | None = None):
"""Register a transaction logfmt callback. """Register a transaction logfmt callback.
@@ -251,12 +346,39 @@ class Kanta(Generic[T]):
return _register return _register
return _register(fn) return _register(fn)
def logemit(self, fn=None):
"""Register a log emitter callback.
Can be used as ``@kanta.logemit``. The callback receives a single
:class:`kanta.logging.LogEvent` describing the event, including the
preferred logger and level, and decides what (if anything) is logged
and where.
A falsy return value marks the event as handled and stops the chain.
A truthy return value passes the event — possibly modified — to the
next registered callback; when all callbacks pass, Kanta renders the
event with its built-in formatting
(:func:`kanta.logging.default_emit`), which a callback may also call
itself to delegate events it does not care about.
"""
def _register(callback):
self._impl.add_logemit(callback)
return callback
if fn is None:
return _register
return _register(fn)
def transaction( def transaction(
self, self,
action: str, action: str,
*, *,
user: str | None = None, user: str | None = None,
extra: Any = None,
mtime: bool | datetime = True, mtime: bool | datetime = True,
log: bool | logging.Logger = True,
logdiff: bool = True,
): ):
"""Create a transactional mutation context manager. """Create a transactional mutation context manager.
@@ -265,12 +387,26 @@ class Kanta(Generic[T]):
user: Optional user identifier stored in metadata and rendered in user: Optional user identifier stored in metadata and rendered in
the log header. Register a ``@kanta.logfmt`` callback to format the log header. Register a ``@kanta.logfmt`` callback to format
the user value; the path ``"$user"`` is passed for this case. the user value; the path ``"$user"`` is passed for this case.
extra: Optional display-only value shown after the action in the
log header. Anything other than ``None`` is printed
str-converted (colored by Kanta), unless a custom
``@kanta.logemit`` handler does something else with it. It is
never persisted in the change record.
mtime: Controls the modification time ``m``. ``True`` (default) mtime: Controls the modification time ``m``. ``True`` (default)
sets ``m`` to the current UTC time. ``False`` omits ``m`` so the sets ``m`` to the current UTC time. ``False`` omits ``m`` so the
previous modification time remains in effect; this is used for previous modification time remains in effect; this is used for
system operations that are not considered modifications. A system operations that are not considered modifications. A
:class:`~datetime.datetime` value sets ``m`` to that explicit :class:`~datetime.datetime` value sets ``m`` to that explicit
time. time.
log: Controls transaction logging. ``True`` (default) uses the
``kanta.transaction`` logger. ``False`` suppresses the
transaction log. A :class:`~logging.Logger` instance writes
output to that logger instead.
logdiff: Whether to build and print the diff body. ``False``
skips diff formatting entirely and logs only the header,
which is useful for large or noisy changesets. Diff output
can also be disabled globally with
``configure_logging(diff=False)``.
Returns: Returns:
A context manager yielding the live state object for mutation. A context manager yielding the live state object for mutation.
@@ -284,5 +420,8 @@ class Kanta(Generic[T]):
self._impl, self._impl,
action, action,
user=user, user=user,
extra=extra,
mtime=mtime, mtime=mtime,
log=log,
logdiff=logdiff,
) )
+279 -37
View File
@@ -6,17 +6,26 @@ import asyncio
import copy import copy
import importlib import importlib
import logging import logging
from datetime import UTC, datetime from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import Any, Generic, TypeVar from typing import Any, Generic, TypeVar
from kanta.callbacks import CallbackRegistry, InjectionContext from kanta.callbacks import CallbackRegistry, InjectionContext, callback_error_reporter
from kanta.exceptions import DatabaseError, DataIntegrityError, ReplayError from kanta.exceptions import DatabaseError, DataIntegrityError, ReplayError
from kanta.migrate import MigrationRegistry from kanta.logging import (
_USER_PATH,
LogEvent,
bootstrap_logger,
emit_event,
migration_logger,
)
from kanta.migrations import MigrationReport, Migrations
from kanta.persistence import PersistenceMixin from kanta.persistence import PersistenceMixin
from kanta.rotation import execute_rotation, plan_rotation
from kanta.serialization import restore_data_in_place, struct_to_dict from kanta.serialization import restore_data_in_place, struct_to_dict
from kanta.serialization.base import replay from kanta.serialization.base import replay
_logger = logging.getLogger(__name__) _logger = logging.getLogger("kanta")
T = TypeVar("T") T = TypeVar("T")
@@ -27,22 +36,27 @@ class KantaImpl(PersistenceMixin, Generic[T]):
def __init__(self, **kwargs: Any): def __init__(self, **kwargs: Any):
self.data_type = kwargs.pop("type") self.data_type = kwargs.pop("type")
self.data: T = kwargs.pop("data") self.data: T = kwargs.pop("data")
self.migrations = kwargs.pop("migrations", None)
self.migration_ctx = kwargs.pop("migration_ctx", None)
self._kanta = kwargs.pop("kanta", None) self._kanta = kwargs.pop("kanta", None)
migrations = kwargs.pop("migrations", None)
retention = kwargs.pop("retention", None)
if isinstance(retention, int) and not isinstance(retention, bool):
retention = timedelta(days=retention)
self.retention: timedelta | None = retention
self.ctx = SimpleNamespace()
super().__init__(**kwargs) super().__init__(**kwargs)
self.migration_registry: MigrationRegistry | None = None self.migrations: Migrations | None = None
if self.migrations is not None: if migrations is not None:
module = ( module = (
importlib.import_module(self.migrations) importlib.import_module(migrations)
if isinstance(self.migrations, str) if isinstance(migrations, str)
else self.migrations else migrations
) )
self.migration_registry = MigrationRegistry.from_module(module) self.migrations = Migrations.from_module(module)
self.in_transaction = False self.in_transaction = False
self.transaction_snapshot: dict[str, Any] | None = None self.transaction_snapshot: dict[str, Any] | None = None
self.opened = False self.opened = False
self.readonly = False
self.bootstrap_action = "bootstrap" self.bootstrap_action = "bootstrap"
self.bootstrap_user: str | None = None self.bootstrap_user: str | None = None
self.bootstrap_mtime: bool | datetime = True self.bootstrap_mtime: bool | datetime = True
@@ -53,9 +67,7 @@ class KantaImpl(PersistenceMixin, Generic[T]):
) )
self.statedict = struct_to_dict(self.data, serializer=self.serializer) self.statedict = struct_to_dict(self.data, serializer=self.serializer)
self.version = ( self.version = self.migrations.dbver if self.migrations is not None else 0
self.migration_registry.dbver if self.migration_registry is not None else 0
)
def add_bootstrap( def add_bootstrap(
self, self,
@@ -75,7 +87,67 @@ class KantaImpl(PersistenceMixin, Generic[T]):
"""Register one transaction logfmt callback.""" """Register one transaction logfmt callback."""
self.callback_registry.register("logfmt", callback, path=path) self.callback_registry.register("logfmt", callback, path=path)
async def open(self, *, create: bool = True) -> None: def add_logmigr(self, callback) -> None:
"""Register one migration logging callback."""
self.callback_registry.register("logmigr", callback)
def add_validate(self, callback) -> None:
"""Register one data validation callback."""
self.callback_registry.register("validate", callback)
def add_logemit(self, callback) -> None:
"""Register one log emitter callback."""
self.callback_registry.register("logemit", callback)
async def _handle_migration_log(
self,
report: MigrationReport,
log: bool | logging.Logger,
) -> None:
"""Route migration logging to callback or default logger."""
assert isinstance(report, MigrationReport)
if self.callback_registry.has("logmigr"):
await self.callback_registry.invoke(
"logmigr",
InjectionContext(
kanta=self._kanta,
report=report,
),
on_error=callback_error_reporter("logmigr"),
)
return
if log is False:
return
migration_log = log if isinstance(log, logging.Logger) else migration_logger
changed = [m for m in report.applied if m.changed]
if not changed:
return
descriptions = [f"{m.name} ({m.description})" for m in changed]
emit_event(
LogEvent(
kind="migrated",
logger=migration_log,
kanta=self._kanta,
filename=str(self.filename),
from_version=report.original,
to_version=report.version,
migrations=descriptions,
),
self.callback_registry.logemit_handlers,
)
async def open(
self,
*,
create: bool = True,
readonly: bool = False,
log: bool | logging.Logger = True,
) -> None:
"""Open the database: load from disk, apply migrations, start background task.""" """Open the database: load from disk, apply migrations, start background task."""
if self.opened: if self.opened:
raise DataIntegrityError( raise DataIntegrityError(
@@ -84,12 +156,17 @@ class KantaImpl(PersistenceMixin, Generic[T]):
action="open", action="open",
) )
self.readonly = readonly
existed_before_open = self.filename.exists() existed_before_open = self.filename.exists()
# Read-only mode never creates the file.
open_create = create and not readonly
content = await asyncio.to_thread( content = await asyncio.to_thread(
self.file.open_and_read, self.file.open_and_read,
self.filename, self.filename,
create=create, create=open_create,
readonly=readonly,
) )
if not create and (not existed_before_open or not content): if not create and (not existed_before_open or not content):
@@ -105,6 +182,12 @@ class KantaImpl(PersistenceMixin, Generic[T]):
action="open", action="open",
) )
# From this point the file is open and must be closed via close().
self.opened = True
if content and self.retention is not None and not readonly:
content = await self._maybe_rotate(content, log)
if content: if content:
try: try:
rr = replay( rr = replay(
@@ -134,42 +217,177 @@ class KantaImpl(PersistenceMixin, Generic[T]):
cause_type=type(e).__name__, cause_type=type(e).__name__,
) from e ) from e
if self.migration_registry is not None: migration_report = None
rr.version = self.migration_registry.apply( state_before_migrations = None
rr.state, rr.version, self.migration_ctx previous_version = rr.version
if self.migrations is not None:
state_before_migrations = copy.deepcopy(rr.state)
migration_report = self.migrations.apply(
rr.state, rr.version, self._kanta
) )
rr.version = migration_report.version
self.statedict = copy.deepcopy(rr.state) migrations_ran = rr.version != previous_version
self.snapshot.ts = (
datetime.fromtimestamp(rr.last_snapshot_mtime, UTC)
if rr.last_snapshot_mtime is not None
else None
)
self.statedict = copy.deepcopy(
state_before_migrations
if state_before_migrations is not None
else rr.state
)
self.data = restore_data_in_place( self.data = restore_data_in_place(
self.data, self.data,
rr.state, rr.state,
self.data_type, self.data_type,
serializer=self.serializer, serializer=self.serializer,
) )
if self.callback_registry.has("validate"):
try:
self.callback_registry.invoke_sync(
"validate",
InjectionContext(data=self.data, kanta=self._kanta),
)
except Exception:
self.opened = False
self.file.close()
raise
self.version = rr.version self.version = rr.version
self.mtime = rr.m self.mtime = rr.m
normalized = struct_to_dict(self.data, serializer=self.serializer) if log is not False and not migrations_ran:
self.queue_change("migrate:msgspec", normalized, mtime=False) logger = log if isinstance(log, logging.Logger) else bootstrap_logger
self.snapshot.ts = ( emit_event(
datetime.fromtimestamp(rr.last_snapshot_mtime, UTC) LogEvent(
if rr.last_snapshot_mtime is not None kind="opened",
else None logger=logger,
) level=logging.DEBUG,
elif self.callback_registry.has("bootstrap"): kanta=self._kanta,
try: filename=str(self.filename.resolve()),
await self.callback_registry.invoke( ),
"bootstrap", self.callback_registry.logemit_handlers,
InjectionContext(data=self.data, kanta=self._kanta),
) )
normalized = struct_to_dict(self.data, serializer=self.serializer)
if self.readonly:
self.statedict = copy.deepcopy(normalized)
else:
# One record per open: migration changes and normalization are
# grouped into migrate:vN, or migrate:msgspec when only the
# serialization drifted.
previous = self.statedict
action = (
f"migrate:v{self.version}" if migrations_ran else "migrate:msgspec"
)
record = self.queue_change(action, normalized, mtime=False)
# The migration summary introduces the diff, so log it first.
if migrations_ran and migration_report is not None:
await self._handle_migration_log(migration_report, log)
if (
record is not None
and log is not False
and not (migrations_ran and self.callback_registry.has("logmigr"))
):
logger = (
log if isinstance(log, logging.Logger) else migration_logger
)
emit_event(
LogEvent(
kind="change",
logger=logger,
level=logging.DEBUG,
kanta=self._kanta,
action=action,
diff=record.diff,
previous=previous,
),
self.callback_registry.logemit_handlers,
)
if migrations_ran or record is not None:
self.snapshot.request_force()
await self.flush()
self.snapshot.maybe_write(
self.file,
self.version,
self.statedict,
m=self.mtime,
now=self.now,
)
elif self.readonly:
self.opened = False
self.file.close()
raise DataIntegrityError(
"Cannot open empty database in read-only mode",
db_path=self.filename,
action="open",
)
else:
try:
if self.callback_registry.has("bootstrap"):
await self.callback_registry.invoke(
"bootstrap",
InjectionContext(data=self.data, kanta=self._kanta),
)
if self.callback_registry.has("validate"):
self.callback_registry.invoke_sync(
"validate",
InjectionContext(data=self.data, kanta=self._kanta),
)
self.statedict = {}
current = struct_to_dict(self.data, serializer=self.serializer) current = struct_to_dict(self.data, serializer=self.serializer)
self.queue_change( record = self.queue_change(
self.bootstrap_action, self.bootstrap_action,
current, current,
user=self.bootstrap_user, user=self.bootstrap_user,
mtime=self.bootstrap_mtime, mtime=self.bootstrap_mtime,
force=True,
) )
if record is not None and log is not False:
logger = (
log if isinstance(log, logging.Logger) else bootstrap_logger
)
emit_event(
LogEvent(
kind="created",
logger=logger,
kanta=self._kanta,
filename=str(self.filename.resolve()),
),
self.callback_registry.logemit_handlers,
)
logfmt = self.callback_registry.build_logfmt(
InjectionContext(
previous_state={},
current_state=current,
kanta=self._kanta,
)
)
formatted_user = self.bootstrap_user
if formatted_user is not None and logfmt is not None:
resolved = logfmt(formatted_user, _USER_PATH)
if resolved is not None:
formatted_user = resolved
emit_event(
LogEvent(
kind="change",
logger=logger,
kanta=self._kanta,
action=self.bootstrap_action,
user=formatted_user,
diff=record.diff,
previous={},
current=current,
logfmt=logfmt,
),
self.callback_registry.logemit_handlers,
)
except Exception: except Exception:
self.opened = False
self.file.close() self.file.close()
try: try:
await asyncio.to_thread(self.filename.unlink, missing_ok=True) await asyncio.to_thread(self.filename.unlink, missing_ok=True)
@@ -177,9 +395,32 @@ class KantaImpl(PersistenceMixin, Generic[T]):
pass pass
raise raise
self.opened = True if not self.readonly:
self.background_task = asyncio.create_task(self._background_loop())
self.background_task = asyncio.create_task(self._background_loop()) async def _maybe_rotate(self, content: bytes, log: bool | logging.Logger) -> bytes:
"""Rotate history older than the retention window (see docs/rotation.md).
Runs while the file is locked and quiescent, before replay. Returns
the (possibly replaced) content to replay. Rotation failures abort the
open with the original file intact.
"""
cutoff = self.now() - self.retention
plan = await asyncio.to_thread(
plan_rotation,
content,
framer=self.framer,
serializer=self.serializer,
cutoff=cutoff,
now=self.now(),
min_diffs=self.snapshot.min_diffs,
)
if plan is None:
return content
await asyncio.to_thread(
execute_rotation, self.filename, self.file, plan, log=log
)
return plan.new_content
async def close(self) -> None: async def close(self) -> None:
"""Stop the background task, flush pending changes, and release the file lock.""" """Stop the background task, flush pending changes, and release the file lock."""
@@ -196,7 +437,8 @@ class KantaImpl(PersistenceMixin, Generic[T]):
# Always run a final flush in case the background task never reached # Always run a final flush in case the background task never reached
# its cancellation handler. # its cancellation handler.
await self.flush() if not self.readonly:
await self.flush()
self.file.close() self.file.close()
self.opened = False self.opened = False
+438 -93
View File
@@ -1,16 +1,41 @@
"""Database change logging with pretty-printed diffs. """Database change logging with pretty-printed diffs.
Provides a logger for JSONL database changes that formats diffs All change-related output is described by a :class:`LogEvent` and dispatched
in a human-readable path.notation style with color coding. through :func:`emit_event`, which runs any registered ``logemit`` callbacks
and falls back to :func:`default_emit` for the built-in formatting. Diff
output is formatted in a human-readable path notation style with color
coding; see :mod:`kanta.tty` for the color palette and line builder.
ANSI codes are stripped at emit time when the standard error stream does
not support color (``NO_COLOR``/``FORCE_COLOR``, tty and journald checks).
""" """
import logging import logging
import re import re
import sys import sys
from collections.abc import Callable from collections.abc import Callable, Iterable
from typing import Any from typing import Any
logger = logging.getLogger("kanta.changes") import msgspec
from kanta.callbacks import describe_callback
from kanta.serialization.base import _apply, unmarshal
from kanta.tty import Line, displaywidth, strip_ansi, use_color
transaction_logger = logging.getLogger("kanta.transaction")
bootstrap_logger = logging.getLogger("kanta.bootstrap")
migration_logger = logging.getLogger("kanta.migration")
# Event loggers carry Kanta-rendered content (colored headers, diffs) and are
# configured at import time; diagnostics from Kanta's internals use the plain
# "kanta" logger so they follow the application's root logging configuration.
EVENT_LOGGERS = ("kanta.bootstrap", "kanta.migration", "kanta.transaction")
# Loggers that emit DEBUG-level events (file-opened summary, migration diffs).
_DEBUG_LOGGERS = ("kanta.bootstrap", "kanta.migration")
_PLAIN_HANDLER_NAME = "kanta.plain"
_logger = logging.getLogger("kanta")
# Pattern to match control characters and bidirectional overrides # Pattern to match control characters and bidirectional overrides
_UNSAFE_CHARS = re.compile( _UNSAFE_CHARS = re.compile(
@@ -21,20 +46,168 @@ _UNSAFE_CHARS = re.compile(
r"]" r"]"
) )
# ANSI color codes
_RESET = "\033[0m"
_SEP = "\033[38;5;242m" # Dark grey for separators
_PATH_PREFIX = "\033[38;5;242m" # Dark grey for path prefix
_PATH_FINAL = "\033[38;5;250m" # Default for final element
_DELETE = "\033[1;31m" # Red for deletions
_ADD = "\033[0;32m" # Green for additions
_ACTION = "\033[1;34m" # Bold blue for action name
_USER = "\033[0;34m" # Blue for user display
# Metadata path used when formatting the transaction actor. # Metadata path used when formatting the transaction actor.
_USER_PATH = "$user" _USER_PATH = "$user"
class LogEvent(msgspec.Struct, kw_only=True):
"""All state describing one loggable event, passed to logemit callbacks.
``kind`` is ``"change"`` (transaction, bootstrap, or migration diff),
``"created"`` (database file created), ``"opened"`` (database file
opened), ``"migrated"`` (migration summary), or ``"aborted"``
(transaction rolled back). ``logger`` and ``level`` are Kanta's
preferred destination; a callback may use them, log elsewhere, or not
log at all.
The event is mutable: a callback may modify it before returning a truthy
value to pass it on, affecting later callbacks and the built-in fallback.
"""
kind: str
logger: logging.Logger
level: int = logging.INFO
kanta: Any = None
action: str | None = None
user: str | None = None
extra: Any = None
error: BaseException | None = None
diff: dict = msgspec.field(default_factory=dict)
previous: dict | None = None
current: dict | None = None
logfmt: Callable[[Any, str], str | None] | None = None
show_diff: bool = True
filename: str | None = None
from_version: int | None = None
to_version: int | None = None
migrations: list[str] = msgspec.field(default_factory=list)
_header: str | None = None
_diff_lines: list[str] | None = None
@property
def header(self) -> str:
"""The default one-line header for this event, built on first access.
Covers every event kind: ``"<action>[ <extra>][ by <user>]"`` for
changes, ``"<action>[ <extra>][ by <user>] transaction aborted:
<error>"`` for aborts, and the ``🛢️ <filename> <verb>`` file
summaries (created / opened / migrated).
"""
if self._header is None:
self._header = self._build_header()
return self._header
@header.setter
def header(self, value: str) -> None:
"""Override the header, keeping the default diff routing.
A logemit callback can restyle the header and return a truthy value:
:func:`default_emit` then logs this header instead of building one.
"""
self._header = value
def _build_header(self) -> str:
if self.kind == "created":
return f"🛢️ {self.filename} created"
if self.kind == "opened":
return f"🛢️ {self.filename} opened"
if self.kind == "migrated":
migrations = ", ".join(self.migrations)
return (
f"🛢️ {self.filename} migrated "
f"v{self.from_version} -> v{self.to_version}: {migrations}"
)
if self.kind == "change":
return format_action_header(self.action or "", self.user, self.extra)
line = Line().action(self.action or "")
if self.extra:
line(" ").target(self.extra)
if self.user:
line(" by ").user(self.user)
line(f" transaction aborted: {self.error}")
return str(line)
@property
def diff_lines(self) -> list[str]:
"""Pretty-printed diff lines, built on first access and cached."""
if self._diff_lines is None:
self._diff_lines = format_diff(self.diff, self.previous, self.logfmt)
return self._diff_lines
def emit_event(
ev: LogEvent,
handlers: Iterable[Callable[[LogEvent], Any]] = (),
*,
fallback: Callable[[LogEvent], None] | None = None,
) -> None:
"""Dispatch *ev* through registered logemit handlers.
Each handler receives the event and may log it (or not) as it sees fit.
A falsy return value stops the chain: the event is considered handled.
A truthy return value passes the event — possibly modified — to the next
handler. When all handlers pass, the *fallback* renders the event;
the default fallback is :func:`default_emit` with the built-in formatting.
Logging must never break functionality: a crashing handler is reported
and the chain falls back to the fallback rendering, and a failure in
the fallback itself is reported and swallowed.
"""
render = fallback if fallback is not None else default_emit
try:
for handler in handlers:
try:
proceed = handler(ev)
except Exception:
_logger.exception(
"Kanta.logemit %s failed, using default formatting",
describe_callback(handler),
)
break
if not proceed:
return
render(ev)
except Exception:
_logger.exception("Kanta failed to emit %s log event", ev.kind)
def _maybe_strip(text: str) -> str:
"""Strip ANSI codes from *text* when stderr has no color support."""
return text if use_color() else strip_ansi(text)
def default_emit(ev: LogEvent) -> None:
"""Emit *ev* with Kanta's built-in formatting.
Logs :attr:`LogEvent.header`; for change events the
:attr:`LogEvent.diff_lines` body follows on the ``<logger>.diff`` child
logger so it can be silenced or routed separately from the headers.
This is what runs when no logemit callback handles the event; custom
callbacks may call it to delegate events they do not care about.
ANSI color codes are stripped after formatting when the standard error
stream does not support color (see :func:`kanta.tty.use_color`).
"""
if ev.kind != "change":
ev.logger.log(ev.level, _maybe_strip(ev.header))
return
diff_logger = logging.getLogger(f"{ev.logger.name}.diff")
lines = ev.diff_lines if ev.show_diff and diff_logger.isEnabledFor(ev.level) else []
if not lines:
ev.logger.log(ev.level, _maybe_strip(ev.header))
return
if len(lines) == 1:
diff_logger.log(ev.level, _maybe_strip(f"{ev.header}{lines[0]}"))
return
ev.logger.log(ev.level, _maybe_strip(ev.header))
for line in lines:
diff_logger.log(ev.level, _maybe_strip(line))
def _join_path(path: str, key: str) -> str: def _join_path(path: str, key: str) -> str:
"""Append *key* to a dot-notation *path*.""" """Append *key* to a dot-notation *path*."""
if not path: if not path:
@@ -42,62 +215,91 @@ def _join_path(path: str, key: str) -> str:
return f"{path}.{key}" return f"{path}.{key}"
def _dim_ellipsis() -> str:
"""Return the truncation ellipsis in the palette's ellipsis color."""
return str(Line().ellipsis(""))
def _format_value( def _format_value(
value: Any, value: Any,
path: str, path: str,
*, *,
max_len: int = 60, max_len: int = 60,
logfmt: Callable[[Any, str], str | None] | None = None, logfmt: Callable[[Any, str], str | None] | None = None,
highlight: Any = None,
) -> str: ) -> str:
"""Format a value for display, truncating if needed.""" """Format a value for display, truncating if needed.
``highlight`` is an optional hook with ``path(text, path)`` and
``value(text, path)`` methods (see :class:`kanta.grep.GrepHighlighter`);
it wraps matched keys and scalar values, and recurses into containers.
"""
if logfmt is not None: if logfmt is not None:
resolved = logfmt(value, path) resolved = logfmt(value, path)
if resolved is not None: if resolved is not None:
if highlight is not None:
resolved = highlight.value(resolved, path)
return resolved return resolved
def keyed(key: Any, key_path: str) -> str:
display = _format_value(key, key_path, max_len=30, logfmt=logfmt)
if highlight is not None:
display = highlight.path(display, key_path)
return display
if value is None: if value is None:
return "null" text = "null"
if isinstance(value, bool): elif isinstance(value, bool):
return "true" if value else "false" text = "true" if value else "false"
if isinstance(value, (int, float)): elif isinstance(value, (int, float)):
return str(value) text = str(value)
if isinstance(value, str): elif isinstance(value, str):
value = _UNSAFE_CHARS.sub("", value) text = _UNSAFE_CHARS.sub("", value)
if len(value) > max_len: if len(text) > max_len:
return value[: max_len - 3] + "..." text = text[: max_len - 1] + _dim_ellipsis()
return value elif isinstance(value, dict):
if isinstance(value, dict):
if not value: if not value:
return "{}" return "{}"
all_true = all(v is True for v in value.values()) all_true = all(v is True for v in value.values())
parts = [] parts = []
for k, v in value.items(): for k, v in value.items():
key_path = _join_path(path, str(k)) key_path = _join_path(path, str(k))
key_display = _format_value(k, key_path, max_len=30, logfmt=logfmt) key_display = keyed(k, key_path)
if all_true: if all_true:
parts.append(key_display) parts.append(key_display)
else: else:
val_display = _format_value(v, key_path, max_len=30, logfmt=logfmt) val_display = _format_value(
v, key_path, max_len=30, logfmt=logfmt, highlight=highlight
)
parts.append(f"{key_display}: {val_display}") parts.append(f"{key_display}: {val_display}")
return "{" + ", ".join(parts) + "}" return "{" + ", ".join(parts) + "}"
if isinstance(value, list): elif isinstance(value, list):
if not value: if not value:
return "[]" return "[]"
parts = [] parts = []
for i, v in enumerate(value): for i, v in enumerate(value):
item_path = _join_path(path, str(i)) item_path = _join_path(path, str(i))
parts.append(_format_value(v, item_path, max_len=30, logfmt=logfmt)) parts.append(
_format_value(
v, item_path, max_len=30, logfmt=logfmt, highlight=highlight
)
)
return "[" + ", ".join(parts) + "]" return "[" + ", ".join(parts) + "]"
text = str(value) else:
if len(text) > max_len: text = str(value)
text = text[: max_len - 3] + "..." if len(text) > max_len:
text = text[: max_len - 1] + _dim_ellipsis()
if highlight is not None:
text = highlight.value(text, path)
return text return text
def _format_path_components( def _format_path_components(
path: list[str], logfmt: Callable[[Any, str], str | None] | None path: list[str],
logfmt: Callable[[Any, str], str | None] | None,
highlight: Any = None,
) -> list[str]: ) -> list[str]:
"""Return path components after applying formatters.""" """Return path components after applying formatters and match highlights."""
if not path: if not path:
return [] return []
result = [] result = []
@@ -108,22 +310,30 @@ def _format_path_components(
resolved = logfmt(component, prefix_path) resolved = logfmt(component, prefix_path)
if resolved is not None: if resolved is not None:
display = resolved display = resolved
if highlight is not None:
display = highlight.path(display, prefix_path)
result.append(display) result.append(display)
return result return result
def _format_path( def _format_path(
path: list[str], logfmt: Callable[[Any, str], str | None] | None path: list[str],
logfmt: Callable[[Any, str], str | None] | None,
final_color: str = "path_final",
highlight: Any = None,
) -> str: ) -> str:
"""Format a path as dot notation with prefix in dark grey, final in default.""" """Format a path as dot notation with prefix in dark grey, final colored.
components = _format_path_components(path, logfmt)
*final_color* names a color in the :data:`kanta.tty.colors` palette.
"""
components = _format_path_components(path, logfmt, highlight)
if not components: if not components:
return "" return ""
if len(components) == 1: line = Line()
return f"{_PATH_FINAL}{components[0]}{_RESET}" if len(components) > 1:
prefix = ".".join(components[:-1]) line.path_prefix(".".join(components[:-1]) + ".")
final = components[-1] getattr(line, final_color)(components[-1])
return f"{_PATH_PREFIX}{prefix}.{_RESET}{_PATH_FINAL}{final}{_RESET}" return str(line)
def _get_nested(data: dict | None, path: list[str]) -> Any: def _get_nested(data: dict | None, path: list[str]) -> Any:
@@ -154,6 +364,13 @@ def _collect_changes(
changes.append(("update" if existed else "add", path, diff)) changes.append(("update" if existed else "add", path, diff))
return return
old_at_path = _get_nested(previous, path)
if isinstance(old_at_path, list):
# List edits ($insert/$delete/per-index) are shown as one whole-list
# update; the diff is already unmarshaled at this point.
changes.append(("update", path, _apply(old_at_path, diff)))
return
for key, value in diff.items(): for key, value in diff.items():
if key == "$delete": if key == "$delete":
if isinstance(value, list): if isinstance(value, list):
@@ -182,7 +399,14 @@ def _collect_changes(
("update" if old_collection is not None else "add", path, value) ("update" if old_collection is not None else "add", path, value)
) )
elif isinstance(key, str) and key.startswith("$"): elif isinstance(key, str) and key.startswith("$"):
changes.append(("add", path, {key: value})) # Unknown $-command or (post-unmarshal) a user key starting with
# "$": treat as a normal key.
new_path = path + [str(key)]
existed = _get_nested(previous, new_path) is not None
if existed:
_collect_changes(value, new_path, changes, previous)
else:
changes.append(("add", new_path, value))
else: else:
new_path = path + [str(key)] new_path = path + [str(key)]
existed = _get_nested(previous, new_path) is not None existed = _get_nested(previous, new_path) is not None
@@ -197,45 +421,67 @@ def _format_change_lines(
path: list[str], path: list[str],
value: Any, value: Any,
logfmt: Callable[[Any, str], str | None] | None = None, logfmt: Callable[[Any, str], str | None] | None = None,
highlight: Any = None,
) -> list[str]: ) -> list[str]:
"""Format a single change as one or more lines.""" """Format a single change as one or more lines."""
path_str = _format_path(path, logfmt=logfmt)
if change_type == "delete": if change_type == "delete":
components = _format_path_components(path, logfmt) components = _format_path_components(path, logfmt, highlight)
if len(components) == 1: line = Line()(" ")
return [f" {_DELETE}{components[0]}{_RESET}"] if len(components) > 1:
prefix = ".".join(components[:-1]) line.path_prefix(".".join(components[:-1]) + ".")
final = components[-1] marker = ""
return [f" {_PATH_PREFIX}{prefix}.{_RESET}{_DELETE}{final}{_RESET}"] if highlight is not None:
marker = highlight.delete(marker, ".".join(path))
line.delete(components[-1], " ", marker)
return [str(line)]
if change_type == "add": if change_type == "add":
path_str = _format_path(path, logfmt, final_color="add", highlight=highlight)
if isinstance(value, dict) and value: if isinstance(value, dict) and value:
lines = [f" {path_str} {_SEP}={_RESET}"] lines = [str(Line()(" ", path_str, " ").sep("="))]
formatted_items = []
base_path = ".".join(path) base_path = ".".join(path)
for k, v in value.items(): keys = []
for k in value:
key_path = _join_path(base_path, str(k)) key_path = _join_path(base_path, str(k))
key_display = _format_value(k, key_path, max_len=30, logfmt=logfmt) key_display = _format_value(k, key_path, max_len=30, logfmt=logfmt)
v_str = _format_value(v, key_path, max_len=30, logfmt=logfmt) if highlight is not None:
key_display = highlight.path(key_display, key_path)
keys.append((k, key_display))
field_width = max(displaywidth(kd) for _, kd in keys)
field_width = max(field_width, 12)
# Each item line is " {key:{field_width}}: {value}"; budget the
# value so the whole line fits in 80 columns.
value_width = max(80 - 4 - field_width - 2, 20)
formatted_items = []
for (k, key_display), v in zip(keys, value.values()):
key_path = _join_path(base_path, str(k))
v_str = _format_value(
v, key_path, max_len=value_width, logfmt=logfmt, highlight=highlight
)
formatted_items.append((key_display, v_str)) formatted_items.append((key_display, v_str))
max_key_len = max(len(k) for k, _ in formatted_items) return lines + [
field_width = max(max_key_len, 12) str(
for k_display, v_str in formatted_items: Line()(" ", k).sep(":")(
padding = " " * (field_width - len(k_display)) " " * (field_width - displaywidth(k)), " ", v
lines.append(f" {k_display}{_SEP}:{_RESET}{padding} {v_str}") )
return lines )
value_str = _format_value(value, ".".join(path), logfmt=logfmt) for k, v in formatted_items
return [f" {path_str} {_SEP}={_RESET} {value_str}"] ]
value_str = _format_value(
value, ".".join(path), logfmt=logfmt, highlight=highlight
)
return [str(Line()(" ", path_str, " ").sep("=")(" ", value_str))]
value_str = _format_value(value, ".".join(path), logfmt=logfmt) value_str = _format_value(value, ".".join(path), logfmt=logfmt, highlight=highlight)
return [f" {path_str} {_SEP}={_RESET} {value_str}"] path_str = _format_path(path, logfmt=logfmt, highlight=highlight)
return [str(Line()(" ", path_str, " ").sep("=")(" ", value_str))]
def format_diff( def format_diff(
diff: dict, diff: dict,
previous: dict | None = None, previous: dict | None = None,
logfmt: Callable[[Any, str], str | None] | None = None, logfmt: Callable[[Any, str], str | None] | None = None,
highlight: Any = None,
) -> list[str]: ) -> list[str]:
"""Format a JSON diff as human-readable lines. """Format a JSON diff as human-readable lines.
@@ -246,26 +492,43 @@ def format_diff(
``path`` is a dot-notation string; ``"$user"`` is used for the ``path`` is a dot-notation string; ``"$user"`` is used for the
transaction actor. If the callable returns ``None``, default transaction actor. If the callable returns ``None``, default
formatting is used. formatting is used.
highlight: Optional match highlighter hook (see
:class:`kanta.grep.GrepHighlighter`) wrapping matched regions.
Returns a list of formatted lines (without newlines). Returns a list of formatted lines (without newlines).
""" """
changes: list[tuple[str, list[str], Any]] = [] changes: list[tuple[str, list[str], Any]] = []
_collect_changes(diff, [], changes, previous) _collect_changes(unmarshal(diff), [], changes, previous)
if not changes: if not changes:
return [] return []
lines = [] lines = []
for change_type, path, value in changes: for change_type, path, value in changes:
lines.extend(_format_change_lines(change_type, path, value, logfmt)) lines.extend(_format_change_lines(change_type, path, value, logfmt, highlight))
return lines return lines
def format_action_header(action: str, user: str | None = None) -> str: def format_action_header(
"""Format the action header line.""" action: str,
action_str = f"{_ACTION}{action}{_RESET}" user: str | None = None,
if user: extra: Any = None,
user_str = f"{_USER}{user}{_RESET}" highlight: Any = None,
return f"{action_str} by {user_str}" ) -> str:
return action_str """Format the default action header line.
``highlight`` is an optional hook with a ``meta(text, field)`` method
(see :class:`kanta.grep.GrepHighlighter`) wrapping matched regions of
the action and user fields.
"""
if highlight is not None:
action = highlight.meta(action, "action")
line = Line().action(action)
if extra is not None and (extra := f"{extra}"):
line(" ").target(extra)
if user is not None and (user := f"{user}"):
if highlight is not None:
user = highlight.meta(user, "user")
line(" by ").user(user)
return str(line)
def log_change( def log_change(
@@ -273,37 +536,119 @@ def log_change(
diff: dict, diff: dict,
user: str | None = None, user: str | None = None,
previous: dict | None = None, previous: dict | None = None,
extra: Any = None,
logfmt: Callable[[Any, str], str | None] | None = None, logfmt: Callable[[Any, str], str | None] | None = None,
*,
logger: logging.Logger = transaction_logger,
level: int = logging.INFO,
log_diff: bool = True,
) -> None: ) -> None:
"""Log a database change with pretty-printed diff. """Log a database change with the built-in formatting.
Compatibility wrapper around :func:`emit_event` with no handlers; Kanta
itself builds a :class:`LogEvent` and dispatches it with the registered
logemit callbacks.
Args: Args:
action: The action name (e.g., "login", "admin:delete_user"). action: The action name (e.g., "login", "admin:delete_user").
diff: The JSON diff dict. diff: The JSON diff dict.
user: Optional already-formatted user name to show in the header. user: Optional already-formatted user name to show in the header.
previous: The previous state dict (for determining add vs update). previous: The previous state dict (for determining add vs update).
extra: Optional display-only value shown after the action in the
header. Anything other than ``None`` is printed str-converted
(colored by Kanta), unless a custom logemit handler does
something else with it.
logfmt: Optional formatter callable ``(value, path) -> str | None``. logfmt: Optional formatter callable ``(value, path) -> str | None``.
logger: Logger to write to. Defaults to the ``kanta.transaction`` logger.
level: Log level to use. Defaults to ``logging.INFO``.
log_diff: Whether to build and emit the diff lines. ``False`` skips
diff formatting entirely and only the header is logged.
""" """
header = format_action_header(action, user) emit_event(
diff_lines = format_diff(diff, previous, logfmt) LogEvent(
kind="change",
if not diff_lines: logger=logger,
logger.info(header) level=level,
return action=action,
user=user,
if len(diff_lines) == 1: extra=extra,
logger.info(f"{header}{diff_lines[0]}") diff=diff,
else: previous=previous,
logger.info(header) logfmt=logfmt,
for line in diff_lines: show_diff=log_diff,
logger.info(line) )
)
def configure_logging() -> None: def _ensure_plain_handler(logger: logging.Logger) -> None:
"""Configure the database logger to output to stderr without prefix.""" """Attach Kanta's no-prefix stderr handler to *logger* if it has none."""
if not logger.handlers: if not logger.handlers:
handler = logging.StreamHandler(sys.stderr) handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(logging.Formatter("%(message)s")) handler.setFormatter(logging.Formatter("%(message)s"))
handler.name = _PLAIN_HANDLER_NAME
logger.addHandler(handler) logger.addHandler(handler)
logger.setLevel(logging.INFO)
logger.propagate = False
def configure_logging(
*,
skiproot: bool = True,
bootstrap: bool = True,
migration: bool = True,
transaction: bool = True,
diff: bool = True,
debug: bool = False,
) -> None:
"""Configure Kanta's default logging output.
Called once at import time with default arguments; call again to change
the toggles. The event loggers ``kanta.bootstrap``, ``kanta.migration``
and ``kanta.transaction`` carry Kanta-rendered output (colored headers,
diffs) and print it bare through a plain stderr handler with
``propagate = False``. Diagnostic messages use the plain ``kanta``
logger and follow the application's root logging configuration.
No levels are set by default: the event loggers inherit the effective
level of the root logger.
Args:
skiproot: If ``True`` (default), event loggers print through Kanta's
own plain handler without propagating to the root logger. If
``False``, Kanta's handler is removed and propagation enabled so
the application's root logger renders event output instead.
bootstrap: Whether bootstrap logs are enabled.
migration: Whether migration logs are enabled.
transaction: Whether transaction logs are enabled.
diff: Whether transaction diff lines are enabled. When ``False``,
only transaction headers are printed and diff formatting is
skipped. Per transaction this is controlled by the ``logdiff``
argument of :meth:`Kanta.transaction`.
debug: Whether to set the event loggers that emit DEBUG-level output
(bootstrap and migration) to ``DEBUG``, revealing output such as
the file-opened summary and migration diffs. ``False`` resets
them to inheriting the root level.
"""
logging.getLogger("kanta.transaction.diff").disabled = not diff
for name, enabled in (
("kanta.bootstrap", bootstrap),
("kanta.migration", migration),
("kanta.transaction", transaction),
):
logging.getLogger(name).disabled = not enabled
for name in _DEBUG_LOGGERS:
logging.getLogger(name).setLevel(logging.DEBUG if debug else logging.NOTSET)
for name in EVENT_LOGGERS:
logger = logging.getLogger(name)
if skiproot:
logger.propagate = False
_ensure_plain_handler(logger)
else:
logger.propagate = True
logger.handlers[:] = [
h for h in logger.handlers if h.name != _PLAIN_HANDLER_NAME
]
configure_logging() # Import-time default setup; call again to reconfigure.
-117
View File
@@ -1,117 +0,0 @@
"""Database schema migration framework.
Migrations are numbered functions discovered automatically via a decorator
or by prefix. Each runs exactly once based on the current version.
"""
from __future__ import annotations
import importlib
import logging
from types import ModuleType
from typing import Any
import msgspec
_logger = logging.getLogger(__name__)
class MigrationCtx(msgspec.Struct, omit_defaults=True):
"""Context passed to each migration function.
Subclass or replace this with your own context type.
"""
pass
class MigrationRegistry:
"""Registry of schema migration functions.
Usage::
registry = MigrationRegistry()
@registry.register
def migrate_v1(d: dict, ctx: MigrationCtx) -> None:
d.setdefault("version", 1)
new_version = registry.apply(state, current_version=0)
Or load from a module::
registry = MigrationRegistry.from_module("myapp.migrations")
new_version = registry.apply(state, current_version=0)
"""
def __init__(self) -> None:
self._migrations: dict[int, Any] = {}
@staticmethod
def _migration_version(fn: Any) -> int:
name = getattr(fn, "__name__", "")
if not name.startswith("migrate_v"):
raise ValueError(f"Invalid migration function name: {name!r}")
suffix = name.removeprefix("migrate_v")
if not suffix.isdigit() or int(suffix) <= 0:
raise ValueError(f"Invalid migration version in function name: {name!r}")
return int(suffix)
def register(self, fn):
"""Decorator to register a migration function."""
version = self._migration_version(fn)
self._migrations[version] = fn
return fn
@classmethod
def from_module(cls, module: str | ModuleType) -> MigrationRegistry:
"""Create a registry by scanning a module for ``migrate_vN`` functions.
Args:
module: A module name (string) or an imported module object.
"""
reg = cls()
if isinstance(module, str):
mod = importlib.import_module(module)
else:
mod = module
for name in dir(mod):
if name.startswith("migrate_v"):
fn = getattr(mod, name)
if callable(fn):
version = reg._migration_version(fn)
reg._migrations[version] = fn
return reg
@property
def dbver(self) -> int:
"""Current schema version (= highest discovered migration, or 0)."""
return max(self._migrations.keys(), default=0)
def apply(
self,
data_dict: dict[str, Any],
current_version: int,
ctx: MigrationCtx | None = None,
*,
silent: bool = False,
) -> int:
"""Apply pending migrations to *data_dict* in place.
Returns the new version after all migrations.
"""
while current_version < self.dbver:
next_version = current_version + 1
fn = self._migrations.get(next_version)
if fn is None:
raise ValueError(
f"Missing migration step migrate_v{next_version} "
f"(highest discovered is v{self.dbver})"
)
fn(data_dict, ctx or MigrationCtx())
current_version = next_version
if not silent:
desc = (fn.__doc__ or fn.__name__).split("\n")[0].rstrip(".")
_logger.info("Applied migration %s: %s", fn.__name__, desc)
return current_version
+196
View File
@@ -0,0 +1,196 @@
"""Database schema migration framework.
Migrations are numbered functions discovered automatically via a decorator
or by prefix. Each runs exactly once based on the current version.
"""
from __future__ import annotations
import copy
import importlib
import inspect
from dataclasses import dataclass
from types import ModuleType
from typing import Any
from kanta.diff import diff
from kanta.exceptions import DatabaseError
# Cache registries by imported module object so that many Kanta instances using
# the same migrations module do not re-scan it each time.
_module_registry_cache: dict[ModuleType, Migrations] = {}
@dataclass
class MigrationInfo:
"""Information about a single migration that ran."""
name: str
description: str
version: int
changed: bool
diff: dict | None = None
before: dict | None = None
@dataclass
class MigrationReport:
"""Report of applying migrations."""
version: int
original: int
applied: list[MigrationInfo]
@property
def migrations(self) -> list[MigrationInfo]:
"""Deprecated alias for :attr:`applied`."""
return self.applied
MigrationResult = MigrationReport # deprecated alias for MigrationReport
class Migrations:
"""Registry of schema migration functions.
Usage::
migrations = Migrations()
@migrations.register
def migrate_v1(d: dict, kanta) -> None:
d.setdefault("version", 1)
kanta.ctx.note = "migrated"
@migrations.register
def migrate_v2(d: dict) -> None:
d.setdefault("version", 2)
report = migrations.apply(state, current_version=0, kanta=kanta)
new_version = report.version
Or load from a module::
migrations = Migrations.from_module("myapp.migrations")
report = migrations.apply(state, current_version=0, kanta=kanta)
"""
def __init__(self) -> None:
self._migrations: dict[int, Any] = {}
@staticmethod
def _migration_version(fn: Any) -> int:
name = getattr(fn, "__name__", "")
if not name.startswith("migrate_v"):
raise ValueError(f"Invalid migration function name: {name!r}")
suffix = name.removeprefix("migrate_v")
if not suffix.isdigit() or int(suffix) <= 0:
raise ValueError(f"Invalid migration version in function name: {name!r}")
return int(suffix)
def register(self, fn):
"""Decorator to register a migration function."""
version = self._migration_version(fn)
self._migrations[version] = fn
return fn
@classmethod
def from_module(cls, module: str | ModuleType) -> Migrations:
"""Create or retrieve a cached registry by scanning a module.
Args:
module: A module name (string) or an imported module object.
"""
if isinstance(module, str):
mod = importlib.import_module(module)
else:
mod = module
try:
return _module_registry_cache[mod]
except KeyError:
pass
reg = cls()
for name in dir(mod):
if name.startswith("migrate_v"):
fn = getattr(mod, name)
if callable(fn):
version = reg._migration_version(fn)
reg._migrations[version] = fn
_module_registry_cache[mod] = reg
return reg
@property
def dbver(self) -> int:
"""Current schema version (= highest discovered migration, or 0)."""
return max(self._migrations.keys(), default=0)
@property
def minver(self) -> int:
"""Minimum supported current version (first migration minus 1, or 0)."""
return min(self._migrations.keys(), default=1) - 1
@staticmethod
def _call_migration(fn: Any, data_dict: dict[str, Any], kanta: Any) -> None:
"""Call *fn* with the data dict and, if accepted, the Kanta instance."""
try:
inspect.signature(fn).bind(data_dict, kanta)
except TypeError:
fn(data_dict)
else:
fn(data_dict, kanta)
def apply(
self,
data_dict: dict[str, Any],
current_version: int,
kanta: Any,
) -> MigrationReport:
"""Apply pending migrations to *data_dict* in place.
Missing intermediate migration steps are silently skipped.
Raises:
DatabaseError: If the database version is newer than the highest
supported version or older than the minimum supported version.
Returns a :class:`MigrationReport` describing the original and new
versions and every migration that ran.
"""
if current_version > self.dbver:
raise DatabaseError(
f"Database version v{current_version} is newer than the "
f"highest supported version v{self.dbver}"
)
if current_version < self.minver:
raise DatabaseError(
f"Database version v{current_version} is older than the "
f"minimum supported version v{self.minver}"
)
migrations: list[MigrationInfo] = []
original = current_version
for version in sorted(self._migrations.keys()):
if version <= current_version:
continue
fn = self._migrations[version]
before = copy.deepcopy(data_dict)
self._call_migration(fn, data_dict, kanta)
current_version = version
changed = before != data_dict
delta = diff(before, data_dict) if changed else None
desc = (fn.__doc__ or f"v{version}").split("\n")[0].rstrip(".")
migrations.append(
MigrationInfo(
name=fn.__name__,
description=desc,
version=version,
changed=changed,
diff=delta,
before=before,
)
)
return MigrationReport(
version=current_version, original=original, applied=migrations
)
+68 -21
View File
@@ -4,14 +4,16 @@ from __future__ import annotations
import asyncio import asyncio
import copy import copy
import inspect
import logging import logging
from collections import deque from collections import deque
from collections.abc import Callable
from datetime import UTC, datetime from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from kanta.callbacks import CallbackRegistry, InjectionContext from kanta.callbacks import CallbackRegistry, InjectionContext, callback_error_reporter
from kanta.diff import compute_diff from kanta.diff import diff
from kanta.exceptions import DatabaseError, DataIntegrityError from kanta.exceptions import DatabaseError, DataIntegrityError
from kanta.filelock import LockedFile from kanta.filelock import LockedFile
from kanta.structs import ChangeRecord from kanta.structs import ChangeRecord
@@ -19,7 +21,7 @@ from kanta.serialization import JsonSerializer, Serializer
from kanta.serialization.framing import Framer from kanta.serialization.framing import Framer
from kanta.snapshot import SnapshotState from kanta.snapshot import SnapshotState
_logger = logging.getLogger(__name__) _logger = logging.getLogger("kanta")
class PersistenceMixin: class PersistenceMixin:
@@ -39,7 +41,9 @@ class PersistenceMixin:
flush_interval: float flush_interval: float
version: int version: int
opened: bool opened: bool
readonly: bool
mtime: datetime | None mtime: datetime | None
clock: Callable[[], datetime] | None
def __init__(self, **kwargs: Any) -> None: def __init__(self, **kwargs: Any) -> None:
"""Initialize persistence-owned state used by mixin methods.""" """Initialize persistence-owned state used by mixin methods."""
@@ -61,6 +65,31 @@ class PersistenceMixin:
self.flush_interval = flush_interval self.flush_interval = flush_interval
self.version = 0 self.version = 0
self.mtime: datetime | None = None self.mtime: datetime | None = None
self.clock: Callable[[], datetime] | None = None
def add_clock(self, callback) -> None:
"""Register a clock callback ``() -> datetime`` replacing the UTC clock."""
if not callable(callback):
raise TypeError("clock callback must be callable")
for param in inspect.signature(callback).parameters.values():
if param.default is inspect.Parameter.empty and param.kind in (
param.POSITIONAL_ONLY,
param.POSITIONAL_OR_KEYWORD,
param.KEYWORD_ONLY,
):
raise TypeError("clock callback must not require arguments")
self.clock = callback
def now(self) -> datetime:
"""Current time from the registered clock (default: UTC now)."""
if self.clock is None:
return datetime.now(UTC)
ts = self.clock()
if not isinstance(ts, datetime):
raise TypeError(
f"clock callback must return a datetime, got {type(ts).__name__}"
)
return ts
def add_fatal_error(self, callback) -> None: def add_fatal_error(self, callback) -> None:
"""Register one fatal error callback in call order.""" """Register one fatal error callback in call order."""
@@ -68,6 +97,8 @@ class PersistenceMixin:
async def _background_loop(self) -> None: async def _background_loop(self) -> None:
"""Background task that periodically flushes changes to disk.""" """Background task that periodically flushes changes to disk."""
if self.readonly:
return
while True: while True:
try: try:
await asyncio.sleep(self.flush_interval) await asyncio.sleep(self.flush_interval)
@@ -79,25 +110,21 @@ class PersistenceMixin:
break break
except DatabaseError as e: except DatabaseError as e:
self.background_error = e self.background_error = e
def _log_callback_error(callback_error, callback):
_logger.exception(
"Background error callback %r failed: %s",
callback,
callback_error,
)
await self.callback_registry.invoke( await self.callback_registry.invoke(
"fatal_error", "fatal_error",
InjectionContext(error=e, kanta=self._kanta), InjectionContext(error=e, kanta=self._kanta),
on_error=_log_callback_error, on_error=callback_error_reporter("fatal_error"),
)
_logger.error(
"Kanta background flush failed; automatic flushing stopped: %s", e
) )
_logger.error("Background flush loop stopped: %s", e)
break break
def maybe_snapshot(self) -> None: def maybe_snapshot(self) -> None:
"""Evaluate and possibly write a snapshot from current state.""" """Evaluate and possibly write a snapshot from current state."""
self.snapshot.maybe_write(self.file, self.version, self.statedict, m=self.mtime) self.snapshot.maybe_write(
self.file, self.version, self.statedict, m=self.mtime, now=self.now
)
def queue_change( def queue_change(
self, self,
@@ -106,6 +133,7 @@ class PersistenceMixin:
*, *,
user: str | None = None, user: str | None = None,
mtime: bool | datetime = True, mtime: bool | datetime = True,
force: bool = False,
) -> ChangeRecord | None: ) -> ChangeRecord | None:
"""Queue a change record internally (thread-safe). """Queue a change record internally (thread-safe).
@@ -118,11 +146,20 @@ class PersistenceMixin:
previous modification time remains in effect; this is used for previous modification time remains in effect; this is used for
system operations that are not considered modifications. A system operations that are not considered modifications. A
:class:`~datetime.datetime` value sets ``m`` to that explicit time. :class:`~datetime.datetime` value sets ``m`` to that explicit time.
force: If ``True``, queue the record even when the diff is empty.
Returns: Returns:
The queued :class:`ChangeRecord`, or ``None`` if the diff was empty. The queued :class:`ChangeRecord`, or ``None`` if the diff was empty
and *force* is ``False``.
""" """
now = datetime.now(UTC) delta = diff(self.statedict, current)
if not delta:
if not force:
return None
delta = {}
# The clock is only read when a record is actually queued.
now = self.now()
if mtime is True: if mtime is True:
m = now m = now
@@ -133,17 +170,13 @@ class PersistenceMixin:
else: else:
raise TypeError("mtime must be True, False, or a datetime") raise TypeError("mtime must be True, False, or a datetime")
diff = compute_diff(self.statedict, current)
if not diff:
return None
record = ChangeRecord( record = ChangeRecord(
ts=now, ts=now,
a=action, a=action,
v=self.version, v=self.version,
u=user, u=user,
m=m, m=m,
diff=diff, diff=delta,
) )
self.pending_changes.append(record) self.pending_changes.append(record)
self.statedict = copy.deepcopy(current) self.statedict = copy.deepcopy(current)
@@ -160,6 +193,13 @@ class PersistenceMixin:
action="flush_sync", action="flush_sync",
) )
if self.readonly:
raise DataIntegrityError(
"Cannot flush in read-only mode",
db_path=self.filename,
action="flush_sync",
)
if self.flush_failed: if self.flush_failed:
return return
@@ -207,6 +247,13 @@ class PersistenceMixin:
action="flush", action="flush",
) )
if self.readonly:
raise DataIntegrityError(
"Cannot flush in read-only mode",
db_path=self.filename,
action="flush",
)
if self.flush_failed: if self.flush_failed:
return return
+378
View File
@@ -0,0 +1,378 @@
"""Line-oriented replay and range selection for kantadb files.
Support machinery for the ``python -m kanta`` CLI: decoding a file into
positioned events, resolving ``-r`` range specifications to line numbers,
replaying state over a line range, and building change log events. Internal
for now; not part of the public API.
"""
from __future__ import annotations
import copy
import dataclasses
from collections.abc import Iterator
from typing import TYPE_CHECKING, Any, Union
import msgspec
from kanta.callbacks import InjectionContext
from kanta.diff import patch
from kanta.exceptions import ReplayError
from kanta.logging import _USER_PATH, LogEvent, transaction_logger
from kanta.structs import ChangeRecord, Snapshot
if TYPE_CHECKING:
from kanta import Kanta
@dataclasses.dataclass
class SnapshotEvent:
"""A snapshot record positioned in the file."""
line_number: int
byte_pos: int
record_index: int
snap: Snapshot
@property
def version(self) -> int:
return self.snap.v
@dataclasses.dataclass
class ChangeEvent:
"""A change record positioned in the file."""
line_number: int
byte_pos: int
record_index: int
record: ChangeRecord
@property
def version(self) -> int:
return self.record.v
Event = Union[SnapshotEvent, ChangeEvent]
class RangeNotFoundError(Exception):
"""A single-item range specification that does not exist in the file.
The message is fully formatted for display, including the offending
input and how many items of that kind the file contains.
"""
@dataclasses.dataclass
class Selection:
"""A resolved range specification.
Either a ``[start_line, end_line)`` line range, or a single snapshot
(``snapshot`` set), used to show the snapshot state without replaying
further records.
"""
start_line: int
end_line: int
snapshot: SnapshotEvent | None = None
def record_label(line_number: int, record_index: int) -> str:
"""Return a padded record label based on line number, falling back to record index."""
number = line_number if line_number else record_index
return f"l{number:<3}"
def scan_events(content: bytes, kanta: Kanta[Any]) -> tuple[list[Event], int]:
"""Decode all records, validating snapshot consistency.
Uses the Kanta instance's serializer and framer. Returns the events in
file order and the number of change records. Raises :class:`ReplayError`
with a located, display-ready message on decode failures or when a
snapshot does not match the replayed state.
"""
impl = kanta._impl
state: dict[str, Any] = {}
events: list[Event] = []
change_count = 0
for is_snapshot, payload, line_number, byte_pos in impl.framer.iter_records(
content, 0
):
record_index = len(events) + 1
label = record_label(line_number, record_index)
try:
if is_snapshot:
snap = impl.serializer.decode(payload, type=Snapshot)
if record_index > 1 and state != snap.state:
raise ReplayError(
f"Snapshot mismatch at {label}: replayed state"
" does not equal the snapshot state.",
line_number=line_number,
byte_pos=byte_pos,
record_type="snapshot",
)
state = snap.state
events.append(SnapshotEvent(line_number, byte_pos, record_index, snap))
else:
record = impl.serializer.decode(payload, type=ChangeRecord)
state = patch(state, record.diff)
events.append(ChangeEvent(line_number, byte_pos, record_index, record))
change_count += 1
except msgspec.DecodeError as exc:
raise ReplayError(
f"Parse error at {label}: {exc}",
line_number=line_number,
byte_pos=byte_pos,
) from exc
return events, change_count
def replay_events(
events: list[Event], end_line: int
) -> Iterator[tuple[Event, dict[str, Any] | None, dict[str, Any]]]:
"""Replay events with line numbers below ``end_line``.
Yields ``(event, previous, state)`` per event: ``previous`` is the state
before a change (``None`` for snapshots) and ``state`` the state after
the event.
"""
state: dict[str, Any] = {}
for event in events:
if event.line_number >= end_line:
break
if isinstance(event, SnapshotEvent):
state = event.snap.state
yield event, None, state
else:
previous = copy.deepcopy(state)
state = patch(state, event.record.diff)
yield event, previous, state
def record_change_event(
record: ChangeRecord,
previous: dict[str, Any],
current: dict[str, Any],
kanta: Kanta[Any],
) -> LogEvent:
"""Build a change :class:`LogEvent` for a replayed record.
The Kanta instance's logfmt callbacks are used for value formatting and
for resolving the user/actor name; the event can then be dispatched with
:func:`kanta.logging.emit_event` and the instance's logemit handlers.
"""
registry = kanta._impl.callback_registry
logfmt = registry.build_logfmt(
InjectionContext(
kanta=kanta,
previous_state=previous,
current_state=current,
)
)
user = record.u
if user is not None:
resolved = logfmt(user, _USER_PATH)
if resolved is not None:
user = resolved
return LogEvent(
kind="change",
logger=transaction_logger,
kanta=kanta,
action=record.a,
user=user,
diff=record.diff,
previous=previous,
logfmt=logfmt,
)
def _plural(count: int, word: str) -> str:
"""Return e.g. ``1 snapshot`` or ``2 snapshots``."""
return f"{count} {word}{'' if count == 1 else 's'}"
def end_of_file(events: list[Event]) -> int:
"""Return the sentinel line number just past the last line of the file."""
return events[-1].line_number + 1 if events else 0
def _change_lines(events: list[Event]) -> list[int]:
"""Return the line numbers of all change records, in file order."""
return [e.line_number for e in events if isinstance(e, ChangeEvent)]
def _snapshot_lines(events: list[Event]) -> list[int]:
"""Return the line numbers addressed by s0, s1, ...
If the file begins with a snapshot, s0 is that snapshot (l1) and s1 is
the next snapshot. Otherwise the file begins with change records (empty
initial state): s0 is l0, the position before the start of the file, and
s1 is the first snapshot.
"""
lines = [e.line_number for e in events if isinstance(e, SnapshotEvent)]
if events and isinstance(events[0], SnapshotEvent):
return lines
return [0, *lines]
def _version_lines(events: list[Event]) -> dict[int, int]:
"""Map each version to the line where it first appears; v0 is l0."""
lines: dict[int, int] = {0: 0}
for event in events:
lines.setdefault(event.version, event.line_number)
return lines
def _event_at_line(events: list[Event], line: int) -> Event | None:
"""Return the event whose file line number exactly matches ``line``."""
for event in events:
if event.line_number == line:
return event
return None
def _parse_bound(bound_str: str) -> tuple[str, int | None]:
"""Parse a range bound with optional unit prefix (l, s, v) or change index."""
if not bound_str:
return "change", None
unit_map = {"l": "line", "s": "snapshot", "v": "version"}
if bound_str[0] in unit_map:
unit = unit_map[bound_str[0]]
rest = bound_str[1:]
if not rest:
raise ValueError(f"empty value in {bound_str!r}")
return unit, int(rest)
return "change", int(bound_str)
def _bound_to_line(
unit: str,
value: int | None,
events: list[Event],
total: int,
is_start: bool,
) -> int:
"""Convert a range bound to a line number.
Out-of-range values are truncated to l0 (before the first line) or to
the line just past the end of the file rather than erroring; a missing
bound means the corresponding file end.
"""
eof = end_of_file(events)
if value is None:
return 0 if is_start else eof
if unit == "change":
lines = _change_lines(events)
if value < 0:
value = total + value
value = max(0, min(value, total))
return lines[value] if value < total else eof
if unit == "line":
if value < 0:
raise ValueError("line numbers do not support negative indexing")
return value
if unit == "snapshot":
lines = _snapshot_lines(events)
idx = len(lines) + value if value < 0 else value
if idx < 0:
return 0
return lines[idx] if idx < len(lines) else eof
if unit == "version":
if value < 0:
raise ValueError("version numbers do not support negative indexing")
return _version_lines(events).get(value, eof)
raise ValueError(f"unknown range unit: {unit}")
def _resolve_range(range_str: str, events: list[Event], total: int) -> tuple[int, int]:
"""Parse a range string into a [start_line, end_line) line range."""
sep = ".." if ".." in range_str else ":"
start_str, end_str = range_str.split(sep, 1)
start_unit, start_val = _parse_bound(start_str)
end_unit, end_val = _parse_bound(end_str)
start_line = _bound_to_line(start_unit, start_val, events, total, is_start=True)
end_line = _bound_to_line(end_unit, end_val, events, total, is_start=False)
# ``..`` makes the end bound inclusive.
if sep == ".." and end_val is not None:
end_line += 1
return min(start_line, end_line), end_line
def _negative_check(unit: str, value: int) -> None:
if value < 0:
raise ValueError(f"{unit} numbers do not support negative indexing")
def select(spec: str, events: list[Event], total: int) -> Selection:
"""Resolve a range specification against the scanned events.
``total`` is the number of change records. Returns a :class:`Selection`:
a line range, or a single snapshot for snapshot selections (``sN``, or
``lN`` pointing at a snapshot). Ranges truncate out-of-bounds values;
a single index must exist and raises :class:`RangeNotFoundError`
otherwise. Syntax errors raise :class:`ValueError`.
"""
if ":" in spec or ".." in spec:
start_line, end_line = _resolve_range(spec, events, total)
return Selection(start_line, end_line)
# A single index must exist; out-of-bounds is an error.
unit, value = _parse_bound(spec)
if value is None:
raise ValueError("single bound must not be empty")
if unit == "snapshot":
lines = _snapshot_lines(events)
idx = len(lines) + value if value < 0 else value
n_snapshots = sum(isinstance(e, SnapshotEvent) for e in events)
count = _plural(n_snapshots, "snapshot")
if not 0 <= idx < len(lines):
raise RangeNotFoundError(f"Snapshot {spec!r} not found in file ({count})")
event = _event_at_line(events, lines[idx])
if event is None:
# s0 with an empty initial state (l0): not a real record, so it
# cannot be selected as a single item.
raise RangeNotFoundError(
f"Snapshot {spec!r} not found in file: the file starts"
f" with an empty initial state ({count})"
)
assert isinstance(event, SnapshotEvent)
return Selection(event.line_number, event.line_number + 1, event)
if unit == "line":
_negative_check(unit, value)
event = _event_at_line(events, value)
if event is None:
n_lines = events[-1].line_number if events else 0
raise RangeNotFoundError(
f"Line {spec!r} not found in file ({_plural(n_lines, 'line')})"
)
if isinstance(event, SnapshotEvent):
return Selection(value, value + 1, event)
return Selection(value, value + 1)
if unit == "version":
_negative_check(unit, value)
lines = _version_lines(events)
if value not in lines:
raise RangeNotFoundError(
f"Version {spec!r} not found in file ({_plural(len(lines), 'version')})"
)
start_line = lines[value]
later = [line for line in lines.values() if line > start_line]
return Selection(start_line, min(later) if later else end_of_file(events))
lines = _change_lines(events)
idx = total + value if value < 0 else value
if not 0 <= idx < total:
raise RangeNotFoundError(
f"Change index {spec!r} not found in file ({_plural(total, 'change')})"
)
end_line = lines[idx + 1] if idx + 1 < total else end_of_file(events)
return Selection(lines[idx], end_line)
+226
View File
@@ -0,0 +1,226 @@
"""Database rotation: bound on-disk history to a retention window.
See docs/rotation.md for the design. All planning happens on the in-memory
bytes of the database file; the caller (KantaImpl.open) performs the actual
copy-aside, in-place rewrite and rotated-file trimming under the file lock.
"""
from __future__ import annotations
import copy
import logging
import shutil
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
from kanta.exceptions import DatabaseError
from kanta.structs import ChangeRecord, Snapshot
from kanta.serialization.base import Serializer, apply_diff
from kanta.serialization.framing import Framer
_logger = logging.getLogger("kanta")
@dataclass
class RotationPlan:
"""Everything needed to execute a rotation on disk."""
new_content: bytes
cutoff_end: int # byte length of the dropped-history prefix
rotated_ts: datetime # ts of the last dropped change record
retained_changes: int
def rotated_path_for(path: Path, ts: datetime) -> Path:
"""Sibling path for the rotated history: ``{stem}@{ISO-basic-ts}.kantadb``.
The timestamp uses ISO 8601 basic format at second precision (e.g.
``20260902T143000Z``); the exact microsecond timestamp remains available
inside the file if ever needed. On collision an incrementing suffix is
inserted before the extension.
"""
stamp = ts.strftime("%Y%m%dT%H%M%SZ")
candidate = path.with_name(f"{path.stem}@{stamp}.kantadb")
n = 1
while candidate.exists():
candidate = path.with_name(f"{path.stem}@{stamp}.{n}.kantadb")
n += 1
return candidate
class _Entry:
"""One parsed record frame with its byte range."""
__slots__ = ("is_snapshot", "record", "byte_pos", "end_pos")
def __init__(
self,
is_snapshot: bool,
record: ChangeRecord | Snapshot,
byte_pos: int,
end_pos: int,
) -> None:
self.is_snapshot = is_snapshot
self.record = record
self.byte_pos = byte_pos
self.end_pos = end_pos
def _scan(content: bytes, *, framer: Framer, serializer: Serializer) -> list[_Entry]:
"""Decode every record in *content* with byte ranges."""
raw = list(framer.iter_records(content, 0))
entries: list[_Entry] = []
for i, (is_snapshot, payload, _line, byte_pos) in enumerate(raw):
end_pos = raw[i + 1][3] if i + 1 < len(raw) else len(content)
record = serializer.decode(
payload, type=Snapshot if is_snapshot else ChangeRecord
)
entries.append(_Entry(is_snapshot, record, byte_pos, end_pos))
return entries
def plan_rotation(
content: bytes,
*,
framer: Framer,
serializer: Serializer,
cutoff: datetime,
now: datetime,
min_diffs: int,
) -> RotationPlan | None:
"""Plan a rotation of *content*, or return None when there is nothing to do.
Raises:
DatabaseError: If replay from the chosen base snapshot does not match
a snapshot found inside the file (corrupt history). Rotation must
be aborted and the original file left untouched.
"""
if not content:
return None
entries = _scan(content, framer=framer, serializer=serializer)
changes = [e for e in entries if not e.is_snapshot]
if not changes:
return None # snapshot-only file: already fully rotated
dropped = [e for e in changes if e.record.ts < cutoff]
if not dropped:
return None # retention window covers all history
retained = [e for e in changes if e.record.ts >= cutoff]
rotated_ts = dropped[-1].record.ts
cutoff_end = retained[0].byte_pos if retained else len(content)
# Replay base: walk snapshots newest-first and take the first (newest)
# one predating the cutoff; fall back to start of file.
base: _Entry | None = None
for e in reversed([e for e in entries if e.is_snapshot]):
if e.record.ts <= cutoff:
base = e
break
state: dict[str, Any] = {}
version = 0
m: datetime | None = None
if base is not None:
snap = base.record
assert isinstance(snap, Snapshot)
state = dict(snap.state)
version = snap.v
m = snap.m
# The cutoff state starts from the replay base: when the base snapshot
# already predates the cutoff, it may itself be the cutoff state.
state_at_cutoff: dict[str, Any] | None = (
copy.deepcopy(state) if base is not None else None
)
version_at_cutoff = version
m_at_cutoff = m
final_version = version
final_m = m
for e in entries:
if base is not None and e.byte_pos <= base.byte_pos:
continue
if e.is_snapshot:
snap = e.record
assert isinstance(snap, Snapshot)
if snap.state != state:
raise DatabaseError(
"rotation aborted: replayed state does not match snapshot "
f"at byte {e.byte_pos}",
action="rotate",
)
if snap.m is not None:
m = snap.m
continue
change = e.record
assert isinstance(change, ChangeRecord)
state = apply_diff(state, change.diff)
version = change.v
if change.m is not None:
m = change.m
if change.ts < cutoff:
state_at_cutoff = copy.deepcopy(state)
version_at_cutoff = version
m_at_cutoff = m
final_version = version
final_m = m
# There is at least one dropped change, so the cutoff state is known.
assert state_at_cutoff is not None
# Build the new content: leading cutoff snapshot, retained changes
# re-framed at fresh offsets, and a final snapshot when enough changes
# survived to warrant one (mirrors the regular snapshot policy).
out = bytearray()
leading = serializer.encode(
Snapshot(
ts=rotated_ts, v=version_at_cutoff, state=state_at_cutoff, m=m_at_cutoff
)
)
out += framer.frame_snapshot(leading, record_offset=0)
for e in retained:
payload = serializer.encode(e.record)
out += framer.frame_change(payload, record_offset=len(out))
if len(retained) >= min_diffs:
closing = serializer.encode(
Snapshot(ts=now, v=final_version, state=state, m=final_m)
)
out += framer.frame_snapshot(closing, record_offset=len(out))
return RotationPlan(
new_content=bytes(out),
cutoff_end=cutoff_end,
rotated_ts=rotated_ts,
retained_changes=len(retained),
)
def execute_rotation(
path: Path, file, plan: RotationPlan, *, log: bool | logging.Logger = True
) -> Path:
"""Execute a planned rotation on disk. Caller must hold the lock on *file*.
1. Copy the original content aside to ``{stem}@{ts}.kantadb``.
2. Rewrite the locked file in place with the new content and fsync.
3. Trim the rotated copy to the dropped-history prefix.
Returns the rotated file path.
"""
rotated = rotated_path_for(path, plan.rotated_ts)
shutil.copy2(path, rotated)
file.replace_content(plan.new_content)
with open(rotated, "r+b") as f:
f.truncate(plan.cutoff_end)
if log:
_logger.info(
"Rotated database %s: kept %d change record(s), "
"moved history before %s to %s",
path,
plan.retained_changes,
plan.rotated_ts.isoformat(),
rotated,
)
return rotated
+87 -20
View File
@@ -121,33 +121,100 @@ def replay(
def _patch_state(state: dict, diff: dict) -> dict: def _patch_state(state: dict, diff: dict) -> dict:
return _apply_diff(state, diff) return apply_diff(state, diff)
def _apply_diff(state: dict, diff: dict) -> dict: def _unescape(value: str) -> str:
"""Reverse jsondiff's ``$$`` escaping; command strings pass through.
Only a ``$$`` prefix is stripped: jsondiff escapes ``$x`` to ``$$x``,
while single ``$`` strings occur verbatim in our own diffs (we do not
escape values) and must be left alone.
"""
if value.startswith("$$"):
return value[1:]
return value
def unmarshal(diff: Any) -> Any:
"""Unescape a marshaled diff (keys, values and ``$delete`` entries).
Needed for jsondiff-produced diffs, which escape ``$``-prefixed values
as well as keys; our own producer escapes keys only, so unescaping
values is a no-op for them.
"""
if isinstance(diff, dict):
return {
_unescape(k) if isinstance(k, str) else k: unmarshal(v)
for k, v in diff.items()
}
if isinstance(diff, list):
return [unmarshal(v) for v in diff]
if isinstance(diff, str):
return _unescape(diff)
return diff
def apply_diff(state: Any, diff: Any) -> Any:
"""Apply a diff.
Understands our own format (plain assignment + ``$delete``) and
jsondiff's marshaled syntax: ``$replace``, positional
``$delete``/``$insert`` and per-index nested diffs on lists. A bare
dict over a non-dict old value is a wholesale replacement.
"""
return _apply(state, unmarshal(diff))
def _is_list_patch(diff: dict) -> bool:
"""Whether a dict diff against a list state is a jsondiff list edit."""
for key in diff:
if key in ("$delete", "$insert"):
continue
try:
int(key)
except (ValueError, TypeError):
return False
return True
def _apply(state: Any, diff: Any) -> Any:
if not isinstance(diff, dict): if not isinstance(diff, dict):
return diff return diff
if not diff:
return state
if "$replace" in diff:
return diff["$replace"]
result = dict(state) if isinstance(state, dict) else state if isinstance(state, list):
if not isinstance(result, dict): if not _is_list_patch(diff):
result = {} # Our own producer replaces a list with a dict (or any other
# type) by plain assignment — no $replace wrapper.
return diff
result = list(state)
deletes = diff.get("$delete")
if deletes:
for pos in deletes:
result.pop(pos)
for pos, value in diff.get("$insert", []):
result.insert(pos, value)
for key, value in diff.items():
if key in ("$delete", "$insert"):
continue
pos = int(key)
result[pos] = _apply(result[pos], value)
return result
result = dict(state) if isinstance(state, dict) else {}
for key, value in diff.items(): for key, value in diff.items():
if key == "$replace":
return value
if key == "$delete": if key == "$delete":
if isinstance(value, list): keys = value if isinstance(value, list) else [value]
for k in value: for k in keys:
result.pop(k, None) result.pop(k, None)
else: elif key == "$insert":
result.pop(value, None)
continue continue
if isinstance(value, dict): elif key in result:
old = result.get(key, {}) result[key] = _apply(result[key], value)
if not isinstance(old, dict): else:
old = {} result[key] = value
result[key] = _apply_diff(old, value)
continue
result[key] = value
return result return result
+24 -11
View File
@@ -3,13 +3,14 @@
from __future__ import annotations from __future__ import annotations
import logging import logging
from collections.abc import Callable
from datetime import UTC, datetime from datetime import UTC, datetime
from kanta.structs import Snapshot from kanta.structs import Snapshot
from kanta.serialization import JsonSerializer, Serializer from kanta.serialization import JsonSerializer, Serializer
from kanta.serialization.framing import Framer, LineFramer from kanta.serialization.framing import Framer, LineFramer
_logger = logging.getLogger(__name__) _logger = logging.getLogger("kanta")
MINDIFFS = 100 MINDIFFS = 100
@@ -34,29 +35,41 @@ class SnapshotState:
"""Force snapshot write on next check.""" """Force snapshot write on next check."""
self._force_pending = True self._force_pending = True
@property
def min_diffs(self) -> int:
"""Minimum accumulated changes before a snapshot may be written."""
return self._min_diffs
def record_changes(self, count: int) -> None: def record_changes(self, count: int) -> None:
self.changes += count self.changes += count
def maybe_write( def maybe_write(
self, file, version: int, state: dict, m: datetime | None = None self,
file,
version: int,
state: dict,
m: datetime | None = None,
now: Callable[[], datetime] | None = None,
) -> None: ) -> None:
"""Write snapshot when thresholds/time policy allows it.""" """Write snapshot when thresholds/time policy allows it."""
if self.changes < self._min_diffs:
return
force = self._force_pending force = self._force_pending
now = datetime.now(UTC) if not force and self.changes < self._min_diffs:
if not force and now.weekday() != 6: # 6 = Sunday
return
sunday_midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
if not force and self.ts is not None and self.ts >= sunday_midnight:
return return
# The clock is only read when a snapshot may actually be written.
ts = now() if now is not None else datetime.now(UTC)
if not force:
if ts.weekday() != 6: # 6 = Sunday
return
sunday_midnight = ts.replace(hour=0, minute=0, second=0, microsecond=0)
if self.ts is not None and self.ts >= sunday_midnight:
return
if not file.is_open: if not file.is_open:
return return
try: try:
self._write(file, version, state, now, m=m) self._write(file, version, state, ts, m=m)
self._force_pending = False self._force_pending = False
except Exception as exc: except Exception as exc:
_logger.error("snapshot: failed to write snapshot: %r", exc) _logger.error("Kanta snapshot failed: %r", exc)
def _write( def _write(
self, file, version: int, state: dict, now: datetime, m: datetime | None = None self, file, version: int, state: dict, now: datetime, m: datetime | None = None
+1 -1
View File
@@ -23,7 +23,7 @@ class ChangeRecord(msgspec.Struct, omit_defaults=True, kw_only=True):
v: int = 0 v: int = 0
u: str | None = None u: str | None = None
m: datetime | None = None m: datetime | None = None
diff: dict diff: dict = {}
class Snapshot(msgspec.Struct, omit_defaults=True): class Snapshot(msgspec.Struct, omit_defaults=True):
+82 -23
View File
@@ -5,14 +5,34 @@ from __future__ import annotations
import logging import logging
from contextlib import contextmanager from contextlib import contextmanager
from datetime import datetime from datetime import datetime
from typing import Any
from kanta.diff import compute_diff from kanta.diff import diff
from kanta.exceptions import DataIntegrityError from kanta.exceptions import DataIntegrityError
from kanta.callbacks import InjectionContext from kanta.callbacks import InjectionContext
from kanta.logging import _USER_PATH, log_change from kanta.logging import _USER_PATH, LogEvent, emit_event, transaction_logger
from kanta.serialization import restore_data_in_place, struct_to_dict from kanta.serialization import restore_data_in_place, struct_to_dict
_logger = logging.getLogger(__name__) _logger = logging.getLogger("kanta")
def _build_logfmt(impl, previous: dict, current: dict):
"""Build the logfmt chain for a state transition."""
return impl.callback_registry.build_logfmt(
InjectionContext(
previous_state=previous,
current_state=current,
kanta=impl._kanta,
)
)
def _resolve_user(logfmt, user: str | None) -> str | None:
"""Resolve *user* for display via the logfmt chain (raw as fallback)."""
if user is None:
return None
resolved = logfmt(user, _USER_PATH)
return resolved if resolved is not None else user
@contextmanager @contextmanager
@@ -21,9 +41,19 @@ def transaction(
action: str, action: str,
*, *,
user: str | None = None, user: str | None = None,
extra: Any = None,
mtime: bool | datetime = True, mtime: bool | datetime = True,
log: bool | logging.Logger = True,
logdiff: bool = True,
): ):
"""Wrap writes in a transaction and yield the live db object.""" """Wrap writes in a transaction and yield the live db object."""
if impl.readonly:
raise DataIntegrityError(
"Cannot start transaction in read-only mode",
db_path=impl.filename,
action=action,
)
if impl.in_transaction: if impl.in_transaction:
raise RuntimeError( raise RuntimeError(
"Nested or simultaneous transactions are not supported " "Nested or simultaneous transactions are not supported "
@@ -35,19 +65,19 @@ def transaction(
if current_dict != impl.statedict: if current_dict != impl.statedict:
is_bootstrap = action in {"bootstrap"} is_bootstrap = action in {"bootstrap"}
if not (is_bootstrap and not impl.statedict): if not (is_bootstrap and not impl.statedict):
diff = compute_diff(impl.statedict, current_dict) delta = diff(impl.statedict, current_dict)
if diff: if delta:
_logger.critical( _logger.critical(
"Database state modified outside of transaction! " "Database state modified outside of transaction! "
"This indicates a bug where changes occurred without a transaction wrapper.\n" "This indicates a bug where changes occurred without a transaction wrapper.\n"
"Changes detected: %s", "Changes detected: %s",
diff, delta,
) )
raise DataIntegrityError( raise DataIntegrityError(
"Database state modified outside of transaction", "Database state modified outside of transaction",
db_path=impl.db_path, db_path=impl.db_path,
action=action, action=action,
diff=diff, diff=delta,
) )
impl.in_transaction = True impl.in_transaction = True
@@ -56,26 +86,55 @@ def transaction(
try: try:
yield impl.data yield impl.data
new_dict = struct_to_dict(impl.data, serializer=impl.serializer) new_dict = struct_to_dict(impl.data, serializer=impl.serializer)
diff = compute_diff(impl.statedict, new_dict) delta = diff(impl.statedict, new_dict)
if diff: if delta:
if impl.callback_registry.has("validate"):
impl.callback_registry.invoke_sync(
"validate",
InjectionContext(data=impl.data, kanta=impl._kanta),
)
previous = impl.statedict previous = impl.statedict
record = impl.queue_change(action, new_dict, user=user, mtime=mtime) record = impl.queue_change(action, new_dict, user=user, mtime=mtime)
if record is not None: if record is not None:
logfmt = impl.callback_registry.build_logfmt( if log is not False:
InjectionContext( logfmt = _build_logfmt(impl, previous, new_dict)
previous_state=previous, logger = (
current_state=new_dict, log if isinstance(log, logging.Logger) else transaction_logger
kanta=impl._kanta,
) )
) emit_event(
formatted_user = user LogEvent(
if user is not None and logfmt is not None: kind="change",
resolved = logfmt(user, _USER_PATH) logger=logger,
if resolved is not None: kanta=impl._kanta,
formatted_user = resolved action=action,
log_change(action, record.diff, formatted_user, previous, logfmt) user=_resolve_user(logfmt, user),
except Exception: extra=extra,
_logger.warning("Transaction '%s' failed, rolling back changes", action) diff=record.diff,
previous=previous,
current=new_dict,
logfmt=logfmt,
show_diff=logdiff,
),
impl.callback_registry.logemit_handlers,
)
except Exception as exc:
resolved_user = None
if user is not None:
logfmt = _build_logfmt(impl, impl.statedict, impl.statedict)
resolved_user = _resolve_user(logfmt, user)
emit_event(
LogEvent(
kind="aborted",
logger=transaction_logger,
level=logging.WARNING,
kanta=impl._kanta,
action=action,
user=resolved_user,
extra=extra,
error=exc,
),
impl.callback_registry.logemit_handlers,
)
if impl.transaction_snapshot is not None: if impl.transaction_snapshot is not None:
impl.data = restore_data_in_place( impl.data = restore_data_in_place(
impl.data, impl.data,
+202
View File
@@ -0,0 +1,202 @@
"""Terminal string building: ANSI colors, display widths, and a line builder.
Colors are stored as bare SGR parameter strings (e.g. ``"1;34"``) without
the ``\\x1b[`` prefix and ``m`` suffix. The :class:`Line` builder understands
how SGR parameters stack: ``0`` clears everything, other parameters apply
sequentially and the last one of each class wins. This lets it emit minimal
escape sequences, folding a needed reset into the same sequence as the next
color instead of emitting a separate one.
"""
from __future__ import annotations
import io
import os
import re
import sys
import unicodedata
from contextlib import suppress
from typing import Any
ESC = "\x1b["
# Matches a full ANSI escape sequence (color codes, cursor movement, ...).
ANSI_RE = re.compile(r"\x1b\[[0-9;:]*[A-Za-z]")
def strip_ansi(text: str) -> str:
"""Remove ANSI escape sequences from *text*."""
return ANSI_RE.sub("", text)
def use_color(stream: io.TextIOBase = sys.stderr) -> bool:
"""Test if the stream supports color codes."""
if os.environ.get("NO_COLOR"): # Non empty means no (no-color.org)
return False
if os.environ.get("FORCE_COLOR", "") not in {"", "0"}: # force-color.org, node
return True
if hasattr(stream, "isatty") and stream.isatty():
return True
with suppress(KeyError, ValueError, OSError): # Journald does color (-ocat)
dev, ino = map(int, os.environ["JOURNAL_STREAM"].split(":", 1))
st = os.fstat(stream.fileno())
return st.st_dev == dev and st.st_ino == ino
return False
def displaywidth(text: str) -> int:
"""Return the terminal column width of *text*, ignoring ANSI sequences.
Wide characters (CJK, most emoji) count as two columns; combining and
zero-width characters count as zero.
"""
return sum(
2
if unicodedata.east_asian_width(c) in "WF"
else 0
if unicodedata.category(c) in ("Mn", "Me", "Cf")
else 1
for c in strip_ansi(text)
)
def pad(text: str, width: int, align: str = "left") -> str:
"""Pad *text* to *width* columns by display width.
*align* is ``"left"`` (padding after), ``"right"`` (padding before), or
``"center"``. Text already at or above *width* is returned unchanged.
"""
missing = width - displaywidth(text)
if missing <= 0:
return text
if align == "right":
return " " * missing + text
if align == "center":
left = missing // 2
return " " * left + text + " " * (missing - left)
return text + " " * missing
class Colors:
"""Kanta's log color palette: bare SGR parameter strings.
Attributes are looked up when a line is rendered, so assignments such as
``colors.action = "36"`` or additions like ``colors.session = "38;5;226"``
take effect immediately, no matter how the object was imported. Added
colors become available on :class:`Line` under the same name.
"""
action = "1;34" # Bold blue for the action name
user = "34" # Blue for the user display
target = "38;5;250" # White for the extra/target display
snapshot = "97" # Bright white for snapshot indicator text
sep = "38;5;242" # Dark grey for separators
path_prefix = "38;5;242" # Dark grey for the leading part of a dotted path
path_final = "38;5;250" # White for the final path element
add = "32" # Green for additions
delete = "1;31" # Bold red for deletions
ellipsis = "38;5;242" # Dark grey for the truncation ellipsis
colors = Colors()
# SGR attribute classes that carry no class siblings (each clears/sets itself).
_ATTR_CLASSES = frozenset({"1", "2", "3", "4", "7", "9"})
def _parse_sgr(spec: str) -> dict[str, str]:
"""Parse a bare SGR parameter string into a ``{class: group}`` state.
Applies the stacking rules: ``0`` clears everything, other parameters
apply sequentially and the last one of each class wins.
"""
state: dict[str, str] = {}
tokens = spec.split(";")
i = 0
while i < len(tokens):
token = tokens[i]
if token == "0":
state.clear()
elif token in ("38", "48"):
cls = "fg" if token == "38" else "bg"
if i + 1 < len(tokens) and tokens[i + 1] == "5":
state[cls] = ";".join(tokens[i : i + 3])
i += 3
continue
if i + 1 < len(tokens) and tokens[i + 1] == "2":
state[cls] = ";".join(tokens[i : i + 4])
i += 4
continue
state[cls] = token
elif token.isdigit() and (30 <= int(token) <= 37 or 90 <= int(token) <= 97):
state["fg"] = token
elif token.isdigit() and (40 <= int(token) <= 47 or 100 <= int(token) <= 107):
state["bg"] = token
elif token in _ATTR_CLASSES:
state[token] = token
else:
state[f"other:{token}"] = token
i += 1
return state
def _sgr_transition(current: dict[str, str], new: dict[str, str]) -> str:
"""Return the minimal escape sequence moving from *current* to *new*."""
if current == new:
return ""
if not new:
return f"{ESC}0m" if current else ""
if not current:
return f"{ESC}{';'.join(new.values())}m"
if current.keys() - new.keys():
# Some attribute must be cleared; fold the reset into one sequence.
return f"{ESC}0;{';'.join(new.values())}m"
changed = [group for cls, group in new.items() if current.get(cls) != group]
return f"{ESC}{';'.join(changed)}m" if changed else ""
class Line:
"""Build a terminal string part by part with colors, width and alignment.
Calling the builder appends content (arguments are converted to ``str``).
Attribute access with a color name arms that palette color for the next
call; the color is reset automatically when that call ends, so a color
always applies to exactly one call::
str(Line().user("Alice")(" by ")) # "Alice" blue, " by " plain
``width`` and ``align`` keyword arguments pad the content of a call by
display width. ``str(line)`` finishes the line, restoring default
colors if any are active.
"""
def __init__(self, palette: Colors | None = None) -> None:
self._palette = palette if palette is not None else colors
self._parts: list[str] = []
self._active: dict[str, str] = {}
self._pending: dict[str, str] = {}
def __getattr__(self, name: str) -> Line:
if name.startswith("_"):
raise AttributeError(name)
spec = getattr(self._palette, name, None)
if spec is None:
raise AttributeError(f"unknown color: {name!r}")
self._pending = _parse_sgr(spec)
return self
def __call__(self, *args: Any, width: int = 0, align: str = "left") -> Line:
text = "".join(str(arg) for arg in args)
if width:
text = pad(text, width, align)
if self._pending != self._active:
self._parts.append(_sgr_transition(self._active, self._pending))
self._active = self._pending
self._parts.append(text)
self._pending = {}
return self
def __str__(self) -> str:
if self._active:
return "".join(self._parts) + f"{ESC}0m"
return "".join(self._parts)
-6
View File
@@ -1,6 +0,0 @@
def main():
print("Hello from kanta!")
if __name__ == "__main__":
main()
+5 -1
View File
@@ -17,20 +17,24 @@ readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [ dependencies = [
"blake3>=1.0.8", "blake3>=1.0.8",
"jsondiff>=2.2.1",
"msgspec>=0.20.0", "msgspec>=0.20.0",
] ]
[project.scripts]
kanta = "kanta.__main__:main"
[project.optional-dependencies] [project.optional-dependencies]
bin = [ bin = [
"blake3>=1.0.8", "blake3>=1.0.8",
] ]
[project.urls] [project.urls]
Homepage = "https://vasanko.com/coders/kanta"
Repository = "https://git.zi.fi/LeoVasanko/kanta" Repository = "https://git.zi.fi/LeoVasanko/kanta"
[dependency-groups] [dependency-groups]
dev = [ dev = [
"jsondiff>=2.2.1",
"pytest>=9.0.2", "pytest>=9.0.2",
"pytest-asyncio>=1.3.0", "pytest-asyncio>=1.3.0",
] ]
+19
View File
@@ -1,3 +1,5 @@
import logging
import pytest import pytest
from kanta.serialization import JsonSerializer, MsgPackSerializer from kanta.serialization import JsonSerializer, MsgPackSerializer
@@ -12,3 +14,20 @@ from kanta.serialization import JsonSerializer, MsgPackSerializer
) )
def format_config(request): def format_config(request):
return request.param return request.param
@pytest.fixture(autouse=True)
def _kanta_event_loggers_propagate():
"""Let kanta's event loggers propagate so caplog captures their records.
Kanta configures them with ``propagate = False`` at import time, which
would hide their records from pytest's root-logger capture handler.
"""
names = ("kanta.bootstrap", "kanta.migration", "kanta.transaction")
loggers = [logging.getLogger(name) for name in names]
previous = [logger.propagate for logger in loggers]
for logger in loggers:
logger.propagate = True
yield
for logger, propagate in zip(loggers, previous):
logger.propagate = propagate
+24 -1
View File
@@ -7,7 +7,7 @@ from uuid import UUID
import msgspec import msgspec
from kanta.kanta import Kanta from kanta.kanta import Kanta
from kanta.structs import ChangeRecord from kanta.structs import ChangeRecord, Snapshot
class User(msgspec.Struct): class User(msgspec.Struct):
@@ -70,6 +70,18 @@ def change_actions(path: Path, format_config) -> list[str]:
return actions return actions
def read_changes(path: Path, format_config) -> list[ChangeRecord]:
_, serializer_cls = format_config
serializer = serializer_cls()
framer = serializer.framer_cls()
records: list[ChangeRecord] = []
for is_snapshot, payload, _, _ in framer.iter_records(path.read_bytes(), 0):
if is_snapshot:
continue
records.append(serializer.decode(payload, type=ChangeRecord))
return records
def make_migrations_module(name: str, fn_name: str, fn): def make_migrations_module(name: str, fn_name: str, fn):
mod = ModuleType(name) mod = ModuleType(name)
mod.__dict__[fn_name] = fn mod.__dict__[fn_name] = fn
@@ -77,6 +89,17 @@ def make_migrations_module(name: str, fn_name: str, fn):
return mod return mod
def read_last_snapshot(path: Path, format_config) -> Snapshot | None:
_, serializer_cls = format_config
serializer = serializer_cls()
framer = serializer.framer_cls()
data = path.read_bytes()
payload, _, _ = framer.scan_last_snapshot(data)
if payload is None:
return None
return serializer.decode(payload, type=Snapshot)
def fixed_change(action: str, diff: dict, *, version: int = 0) -> ChangeRecord: def fixed_change(action: str, diff: dict, *, version: int = 0) -> ChangeRecord:
return ChangeRecord( return ChangeRecord(
ts=datetime(2026, 1, 1, tzinfo=UTC), a=action, v=version, diff=diff ts=datetime(2026, 1, 1, tzinfo=UTC), a=action, v=version, diff=diff
+160 -14
View File
@@ -1,8 +1,8 @@
from typing import Any from typing import Any, Optional, Union
import pytest import pytest
from kanta import Kanta from kanta import DictPrev, DictState, Kanta
from kanta.callbacks import DictPost, DictPre, LogFmt from kanta.callbacks import DictPost, DictPre, LogFmt
from kanta.exceptions import DatabaseError from kanta.exceptions import DatabaseError
@@ -49,16 +49,60 @@ def test_logfmt_requires_value_annotation(tmp_path, format_config):
return None return None
def test_logfmt_requires_return_annotation(tmp_path, format_config): def test_logfmt_allows_missing_return_annotation(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config) kanta = make_kanta(tmp_path / "test.db", Data, format_config)
with pytest.raises(TypeError, match="must annotate its return"): @kanta.logfmt
def resolve_names(value: str, current: DictPost):
return None
@kanta.logfmt
def resolve_names(value: str, current: DictPost): def test_logfmt_class_allows_missing_return_annotation(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config)
@kanta.logfmt
class UserLogFmt(LogFmt):
def resolve(self, value: str, path: str):
return None return None
# fmt: off
def test_logfmt_accepts_optional_return_typing_forms(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config)
@kanta.logfmt
def resolve_optional(value: str) -> Optional[str]: # noqa: UP007
return value
@kanta.logfmt
def resolve_union(value: str) -> Union[str, None]: # noqa: UP007
return value
@kanta.logfmt
def resolve_pipe(value: "str") -> "str | None":
return value
def test_logfmt_class_accepts_optional_return_typing_forms(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config)
@kanta.logfmt
class OptionalStyle(LogFmt):
def resolve(self, value: str, path: str) -> Optional[str]: # noqa: UP007
return value
@kanta.logfmt
class UnionStyle(LogFmt):
def resolve(self, value: str, path: str) -> Union[str, None]: # noqa: UP007
return value
@kanta.logfmt
class StringStyle(LogFmt):
def resolve(self, value: "str", path: "str") -> "str | None":
return value
# fmt: on
def test_logfmt_rejects_async_callback(tmp_path, format_config): def test_logfmt_rejects_async_callback(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config) kanta = make_kanta(tmp_path / "test.db", Data, format_config)
@@ -104,7 +148,7 @@ async def test_bootstrap_injects_kanta(tmp_path, format_config):
async def test_logfmt_injects_states(tmp_path, format_config, caplog): async def test_logfmt_injects_states(tmp_path, format_config, caplog):
import logging import logging
caplog.set_level(logging.INFO, logger="kanta.changes") caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db" path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config) kanta = make_kanta(path, Data, format_config)
@@ -123,16 +167,118 @@ async def test_logfmt_injects_states(tmp_path, format_config, caplog):
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_logfmt_class_injection(tmp_path, format_config, caplog): async def test_logfmt_injects_states_by_name(tmp_path, format_config, caplog):
import logging import logging
caplog.set_level(logging.INFO, logger="kanta.changes") caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
@kanta.logfmt
def resolve_users(value: str, prev, state: dict | None) -> str | None:
assert prev == {}
assert state is not None
return state.get("users", {}).get(value, {}).get("name")
await kanta.open()
with kanta.transaction(action="create_user") as data:
data.users["uuid-9"] = User(name="Carol")
await kanta.close()
assert "Carol" in caplog.text
@pytest.mark.asyncio
async def test_logfmt_state_name_ignores_annotation(tmp_path, format_config, caplog):
import logging
caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
# Matching by name does not check the annotation.
@kanta.logfmt
def resolve_users(value: str, state: int) -> str | None:
return state.get("users", {}).get(value, {}).get("name")
await kanta.open()
with kanta.transaction(action="create_user") as data:
data.users["uuid-10"] = User(name="Dave")
await kanta.close()
assert "Dave" in caplog.text
@pytest.mark.asyncio
async def test_logfmt_tag_takes_precedence_over_name(tmp_path, format_config, caplog):
import logging
caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
@kanta.logfmt
def check_prev(value: str, anything: DictPrev) -> str | None:
assert anything == {}
return None
@kanta.logfmt
def resolve_users(value: str, prev: DictState) -> str | None:
# The tag wins: prev receives the current state despite its name.
return prev.get("users", {}).get(value, {}).get("name")
await kanta.open()
with kanta.transaction(action="create_user") as data:
data.users["uuid-11"] = User(name="Erin")
await kanta.close()
assert "Erin" in caplog.text
@pytest.mark.asyncio
async def test_logfmt_class_state_attribute(tmp_path, format_config, caplog):
import logging
caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db" path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config) kanta = make_kanta(path, Data, format_config)
@kanta.logfmt @kanta.logfmt
class UserLogFmt(LogFmt): class UserLogFmt(LogFmt):
def resolve(self, value: str, path: str) -> str | None: def resolve(self, value: str, path: str) -> str | None:
if not isinstance(value, str):
return None
return self.state.get("users", {}).get(value, {}).get("name")
await kanta.open()
with kanta.transaction(action="create_user") as data:
data.users["uuid-12"] = User(name="Fred")
await kanta.close()
assert "Fred" in caplog.text
@pytest.mark.asyncio
async def test_logfmt_class_injection(tmp_path, format_config, caplog):
import logging
caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
@kanta.logfmt
class UserLogFmt(LogFmt):
def resolve(self, value: str, path: str) -> str | None:
if not isinstance(value, str):
return None
return self.current_state.get("users", {}).get(value, {}).get("name") return self.current_state.get("users", {}).get(value, {}).get("name")
await kanta.open() await kanta.open()
@@ -149,7 +295,7 @@ async def test_logfmt_class_injection(tmp_path, format_config, caplog):
async def test_multiple_logfmt_chain(tmp_path, format_config, caplog): async def test_multiple_logfmt_chain(tmp_path, format_config, caplog):
import logging import logging
caplog.set_level(logging.INFO, logger="kanta.changes") caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db" path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config) kanta = make_kanta(path, Data, format_config)
@@ -177,7 +323,7 @@ async def test_multiple_logfmt_chain(tmp_path, format_config, caplog):
async def test_logfmt_path_context(tmp_path, format_config, caplog): async def test_logfmt_path_context(tmp_path, format_config, caplog):
import logging import logging
caplog.set_level(logging.INFO, logger="kanta.changes") caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db" path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config) kanta = make_kanta(path, Data, format_config)
@@ -201,7 +347,7 @@ async def test_logfmt_path_context(tmp_path, format_config, caplog):
async def test_logfmt_decorator_path_filters_calls(tmp_path, format_config, caplog): async def test_logfmt_decorator_path_filters_calls(tmp_path, format_config, caplog):
import logging import logging
caplog.set_level(logging.INFO, logger="kanta.changes") caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db" path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config) kanta = make_kanta(path, Data, format_config)
@@ -227,7 +373,7 @@ async def test_logfmt_decorator_path_filters_calls(tmp_path, format_config, capl
async def test_logfmt_user_path_replaces_user_display(tmp_path, format_config, caplog): async def test_logfmt_user_path_replaces_user_display(tmp_path, format_config, caplog):
import logging import logging
caplog.set_level(logging.INFO, logger="kanta.changes") caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db" path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config) kanta = make_kanta(path, Data, format_config)
@@ -249,7 +395,7 @@ async def test_logfmt_user_path_replaces_user_display(tmp_path, format_config, c
async def test_logfmt_non_string_value(tmp_path, format_config, caplog): async def test_logfmt_non_string_value(tmp_path, format_config, caplog):
import logging import logging
caplog.set_level(logging.INFO, logger="kanta.changes") caplog.set_level(logging.INFO, logger="kanta.transaction")
path = tmp_path / "test.db" path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config) kanta = make_kanta(path, Data, format_config)
+174
View File
@@ -0,0 +1,174 @@
"""Tests for the ``python -m kanta`` CLI output formatting."""
import sys
from datetime import UTC, datetime
import pytest
from kanta.__main__ import (
_extra_import_paths,
_format_ts,
_import_dotted,
_import_kanta_object,
main,
)
from kanta.serialization import JsonSerializer
from kanta.serialization.framing import LineFramer
from kanta.structs import ChangeRecord, Snapshot
def test_format_ts_strips_microseconds():
"""Timestamps are rendered without microsecond precision."""
dt = datetime(2026, 8, 12, 10, 6, 52, 375398, tzinfo=UTC)
assert _format_ts(dt) == "2026-08-12 10:06:52"
def test_extra_import_paths_are_temporary(tmp_path, monkeypatch):
"""CWD and nearby venv site-packages are added only for the import block."""
parent_dir = tmp_path / "parent"
cwd = parent_dir / "child"
venv_site = (
cwd
/ ".venv"
/ "lib"
/ f"python{sys.version_info.major}.{sys.version_info.minor}"
/ "site-packages"
)
venv_site.mkdir(parents=True)
parent_venv_site = (
parent_dir
/ ".venv"
/ "lib"
/ f"python{sys.version_info.major}.{sys.version_info.minor}"
/ "site-packages"
)
parent_venv_site.mkdir(parents=True)
monkeypatch.chdir(cwd)
cwd_str = str(cwd)
venv = str(venv_site)
parent_venv = str(parent_venv_site)
before = sys.path.copy()
with _extra_import_paths():
during = sys.path.copy()
assert cwd_str in during
assert venv in during
assert parent_venv in during
assert during.index(cwd_str) < during.index(venv) < during.index(parent_venv)
assert sys.path == before
def test_extra_import_paths_ignores_other_python_versions(tmp_path, monkeypatch):
"""Only the site-packages for the running Python version is picked up."""
current_site = (
tmp_path
/ ".venv"
/ "lib"
/ f"python{sys.version_info.major}.{sys.version_info.minor}"
/ "site-packages"
)
other_site = tmp_path / ".venv" / "lib" / "python9.9" / "site-packages"
current_site.mkdir(parents=True)
other_site.mkdir(parents=True)
monkeypatch.chdir(tmp_path)
with _extra_import_paths():
assert str(current_site) in sys.path
assert str(other_site) not in sys.path
def test_cli_snapshot_line_format(tmp_path, capsys, monkeypatch):
"""Snapshot lines are timestamped and colored with metadata."""
monkeypatch.setenv("FORCE_COLOR", "1")
monkeypatch.delenv("NO_COLOR", raising=False)
path = tmp_path / "test.kantadb"
ts = datetime(2026, 8, 12, 10, 6, 52, 375398, tzinfo=UTC)
mtime = datetime(2026, 8, 12, 9, 0, 0, tzinfo=UTC)
serializer = JsonSerializer()
framer = LineFramer()
snapshot = Snapshot(ts=ts, v=1, m=mtime, state={"counter": 5})
change = ChangeRecord(ts=ts, a="inc", v=1, u="user1", diff={"counter": 6})
data = framer.frame_snapshot(
serializer.encode(snapshot), record_offset=0
) + framer.frame_change(serializer.encode(change), record_offset=0)
path.write_bytes(data)
code = main([str(path)])
assert code == 0
err = capsys.readouterr().err
# No microsecond precision anywhere.
assert "10:06:52" in err
assert "10:06:52.375398" not in err
# Snapshot line: bright white snapshot/sN, white version/mtime, dark size.
assert "\x1b[97msnapshot s0" in err
assert "\x1b[38;5;250m v1 2026-08-12 09:00:00" in err
assert "\x1b[38;5;242m 13 B" in err
def test_cli_strips_ansi_without_color_support(tmp_path, capsys, monkeypatch):
"""Without a tty and with NO_COLOR set, output contains no ANSI codes."""
monkeypatch.setenv("NO_COLOR", "1")
monkeypatch.delenv("FORCE_COLOR", raising=False)
path = tmp_path / "test.kantadb"
ts = datetime(2026, 8, 12, 10, 6, 52, 375398, tzinfo=UTC)
serializer = JsonSerializer()
framer = LineFramer()
snapshot = Snapshot(ts=ts, v=1, m=None, state={"counter": 5})
change = ChangeRecord(ts=ts, a="inc", v=1, u="user1", diff={"counter": 6})
data = framer.frame_snapshot(
serializer.encode(snapshot), record_offset=0
) + framer.frame_change(serializer.encode(change), record_offset=0)
path.write_bytes(data)
code = main([str(path)])
assert code == 0
err = capsys.readouterr().err
assert "\x1b[" not in err
assert "snapshot s0" in err
def test_cli_version_on_help_and_version_flag(capsys):
"""--help and --version print the installed package version."""
import importlib.metadata
version = importlib.metadata.version("kanta")
with pytest.raises(SystemExit) as help_exit:
main(["--help"])
assert help_exit.value.code == 0
assert f"kanta {version}" in capsys.readouterr().out
with pytest.raises(SystemExit) as version_exit:
main(["--version"])
assert version_exit.value.code == 0
assert capsys.readouterr().out.strip() == f"kanta {version}"
def test_import_dotted_from_file_path(tmp_path):
"""--data can be a filesystem path with an optional colon-separated symbol."""
module = tmp_path / "models.py"
module.write_text("class Data:\n pass\n")
result = _import_dotted(f"{module}:Data")
assert result.__name__ == "Data"
def test_import_kanta_object_from_file_path(tmp_path):
"""--kanta can be a filesystem path; default symbol is ``kanta``."""
module = tmp_path / "database.py"
module.write_text("class Kanta:\n pass\nkanta = Kanta()\n")
result = _import_kanta_object(str(module))
assert type(result).__name__ == "Kanta"
def test_import_kanta_object_from_file_path_with_symbol(tmp_path):
"""--kanta can be a filesystem path with an explicit colon-separated symbol."""
module = tmp_path / "database.py"
module.write_text("class CustomKanta:\n pass\nmy_kanta = CustomKanta()\n")
result = _import_kanta_object(f"{module}:my_kanta")
assert type(result).__name__ == "CustomKanta"
+135
View File
@@ -0,0 +1,135 @@
from datetime import UTC, datetime, timedelta
import pytest
from .support import (
Data,
make_kanta,
make_migrations_module,
read_changes,
read_last_snapshot,
)
T0 = datetime(2026, 1, 1, 12, 0, tzinfo=UTC)
def test_clock_rejects_non_callable(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config)
with pytest.raises(TypeError, match="must be callable"):
kanta.clock(42)
def test_clock_rejects_required_argument(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config)
with pytest.raises(TypeError, match="must not require arguments"):
@kanta.clock
def fake_now(tz) -> datetime:
return T0
@pytest.mark.asyncio
async def test_clock_rejects_non_datetime_result(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config)
@kanta.clock
def fake_now() -> datetime:
return "noon"
with pytest.raises(TypeError, match="must return a datetime"):
await kanta.open(log=False)
@pytest.mark.asyncio
async def test_clock_controls_record_timestamps(tmp_path, format_config):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
current = T0
@kanta.clock
def fake_now() -> datetime:
return current
await kanta.open(log=False)
current = T0 + timedelta(hours=1)
with kanta.transaction(action="update") as data:
data.counter = 1
current = T0 + timedelta(hours=2)
with kanta.transaction(action="repair", mtime=False) as data:
data.counter = 2
await kanta.close()
bootstrap, update, repair = read_changes(path, format_config)
assert bootstrap.ts == T0
assert bootstrap.m == T0
assert update.ts == T0 + timedelta(hours=1)
assert update.m == T0 + timedelta(hours=1)
# System operation: stamped by the clock, but m is not updated.
assert repair.ts == T0 + timedelta(hours=2)
assert repair.m is None
assert kanta.mtime == T0 + timedelta(hours=1)
@pytest.mark.asyncio
async def test_clock_not_read_without_changes(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config)
reads = 0
@kanta.clock
def fake_now() -> datetime:
nonlocal reads
reads += 1
return T0
await kanta.open(log=False) # bootstrap record: one read
reads = 0
with kanta.transaction(action="noop"):
pass # no changes, no record, no clock read
await kanta.close() # no snapshot written, no clock read
assert reads == 0
@pytest.mark.asyncio
async def test_clock_controls_migration_and_snapshot_timestamps(
tmp_path, format_config
):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
@kanta.clock
def fake_now() -> datetime:
return T0
await kanta.open(log=False)
await kanta.close()
def migrate_v1(d):
"""Bump counter"""
d["counter"] = 1
migrations = make_migrations_module("clock_migrations", "migrate_v1", migrate_v1)
t1 = T0 + timedelta(days=1)
kanta2 = make_kanta(path, Data, format_config, migrations=migrations)
@kanta2.clock
def fake_now2() -> datetime:
return t1
await kanta2.open(log=False)
await kanta2.close()
migrate_records = [
r for r in read_changes(path, format_config) if r.a.startswith("migrate:")
]
assert migrate_records
assert all(r.ts == t1 for r in migrate_records)
snapshot = read_last_snapshot(path, format_config)
assert snapshot is not None
assert snapshot.ts == t1
# mtime is carried forward from the last real modification.
assert snapshot.m == T0
+249 -7
View File
@@ -1,16 +1,258 @@
from kanta.diff import compute_diff """Tests for our own diff producer/consumer and jsondiff compatibility.
jsondiff is a dev dependency used only here, to verify that:
- jsondiff.patch(..., marshal=True) can apply patches produced by
diff (our format is a subset of jsondiff's marshaled syntax);
- apply_diff can apply patches produced by jsondiff.diff(..., marshal=True),
including positional $insert/$delete list edits and per-index nested diffs.
"""
import jsondiff
import pytest
from kanta.diff import diff, patch
from kanta.logging import format_diff
from kanta.serialization.base import apply_diff
# --- Producer: diff ------------------------------------------------
def test_no_diff(): def test_no_diff():
assert compute_diff({"a": 1}, {"a": 1}) is None assert diff({"a": 1}, {"a": 1}) is None
assert diff({}, {}) is None
def test_simple_diff(): def test_simple_diff():
diff = compute_diff({"a": 1}, {"a": 2}) delta = diff({"a": 1}, {"a": 2})
assert diff is not None assert delta is not None
assert diff == {"a": 2} assert delta == {"a": 2}
def test_nested_diff(): def test_nested_diff():
diff = compute_diff({"x": {"y": 1}}, {"x": {"y": 2}}) delta = diff({"x": {"y": 1}}, {"x": {"y": 2}})
assert diff == {"x": {"y": 2}} assert delta == {"x": {"y": 2}}
def test_key_added():
assert diff({"a": 1}, {"a": 1, "b": 2}) == {"b": 2}
def test_key_removed():
assert diff({"a": 1, "b": 2}, {"a": 1}) == {"$delete": ["b"]}
def test_last_key_removed_is_delete_not_replace():
# jsondiff's minimal-diff search emits {"$replace": {}} here; we emit
# what actually happened: the key was deleted.
assert diff({"a": 1}, {}) == {"$delete": ["a"]}
assert diff({"x": {"y": 1}}, {"x": {}}) == {"x": {"$delete": ["y"]}}
def test_list_changes_are_full_assignment():
# No $insert/$delete positional edits: lists are replaced wholesale.
assert diff({"l": [1, 2]}, {"l": [1, 2, 3]}) == {"l": [1, 2, 3]}
assert diff({"l": [1, 2, 3]}, {"l": [1, 3]}) == {"l": [1, 3]}
assert diff({"l": [1]}, {"l": []}) == {"l": []}
def test_list_with_unchanged_prefix_is_full_assignment():
delta = diff({"l": ["a", "b", "c"]}, {"l": ["a", "x", "b", "c"]})
assert delta == {"l": ["a", "x", "b", "c"]}
def test_type_changes_are_full_assignment():
assert diff({"a": {"x": 1}}, {"a": [1]}) == {"a": [1]}
# A dict replacing a non-dict is a plain assignment too: the consumer
# sees from the old value whether to patch (dict) or replace.
assert diff({"a": [1]}, {"a": {"x": 1}}) == {"a": {"x": 1}}
assert diff({"a": 1}, {"a": None}) == {"a": None}
def test_new_dict_value_assigned_wholesale():
assert diff({}, {"a": {"x": 1}}) == {"a": {"x": 1}}
def test_dollar_keys_escaped():
assert diff({}, {"$weird": 1}) == {"$$weird": 1}
assert diff({"$weird": 1}, {"$weird": 2}) == {"$$weird": 2}
assert diff({"$weird": 1}, {}) == {"$delete": ["$$weird"]}
def test_dollar_values_not_escaped():
# Only keys are escaped; values are stored verbatim, even "$delete".
assert diff({"s": 1}, {"s": "$y"}) == {"s": "$y"}
assert diff({"s": 1}, {"s": "$delete"}) == {"s": "$delete"}
assert diff({}, {"o": {"s": "$y", "l": ["$z"]}}) == {
"o": {"s": "$y", "l": ["$z"]}
}
# --- Consumer: apply_diff / patch -------------------------------------
def test_patch_delegates():
assert patch({"a": 1}, {"a": 2}) == {"a": 2}
def test_apply_scalar_and_add():
assert apply_diff({"a": 1}, {"a": 2, "b": 3}) == {"a": 2, "b": 3}
def test_apply_delete():
assert apply_diff({"a": 1, "b": 2}, {"$delete": ["b"]}) == {"a": 1}
assert apply_diff({"a": 1}, {"$delete": ["a"]}) == {}
def test_apply_replace():
assert apply_diff({"a": 1, "b": 2}, {"$replace": {"c": 3}}) == {"c": 3}
assert apply_diff({"x": {"a": 1}}, {"x": {"$replace": [1]}}) == {"x": [1]}
def test_apply_nested_delete():
diff = {"x": {"$delete": ["y"]}}
assert apply_diff({"x": {"y": 2, "z": 3}}, diff) == {"x": {"z": 3}}
def test_apply_list_insert():
diff = {"l": {"$insert": [[1, "x"]]}}
assert apply_diff({"l": ["a", "b"]}, diff) == {"l": ["a", "x", "b"]}
def test_apply_list_delete():
diff = {"l": {"$delete": [1]}}
assert apply_diff({"l": ["a", "b", "c"]}, diff) == {"l": ["a", "c"]}
def test_apply_list_delete_multiple_positions():
# jsondiff emits positions in descending order for sequential pops.
diff = {"l": {"$delete": [4, 2, 0]}}
assert apply_diff({"l": [0, 1, 2, 3, 4]}, diff) == {"l": [1, 3]}
def test_apply_list_insert_and_delete():
diff = {"l": {"$insert": [[0, 9], [2, 8], [4, 7]], "$delete": [2, 0]}}
assert apply_diff({"l": [0, 1, 2, 3]}, diff) == {"l": [9, 1, 8, 3, 7]}
def test_apply_list_per_index_nested_diff():
diff = {"l": {"1": {"y": 3}}}
state = {"l": [{"x": 1}, {"y": 2}]}
assert apply_diff(state, diff) == {"l": [{"x": 1}, {"y": 3}]}
def test_apply_escaped_keys_and_values():
assert apply_diff({}, {"$$weird": 1}) == {"$weird": 1}
assert apply_diff({"$weird": 1}, {"$delete": ["$$weird"]}) == {}
# jsondiff escapes $-values as "$$.."; those are unescaped on apply.
assert apply_diff({"s": 1}, {"s": "$$y"}) == {"s": "$y"}
assert apply_diff({"s": 1}, {"s": "$$delete"}) == {"s": "$delete"}
# Our own producer stores values verbatim; single-$ stays as-is.
assert apply_diff({"s": 1}, {"s": "$y"}) == {"s": "$y"}
assert apply_diff({"s": 1}, {"s": "$delete"}) == {"s": "$delete"}
assert apply_diff({}, {"o": {"s": "$$y", "l": ["$$z"]}}) == {
"o": {"s": "$y", "l": ["$z"]}
}
def test_apply_empty_diff():
assert apply_diff({"a": 1}, {}) == {"a": 1}
def test_apply_diff_on_missing_state():
assert apply_diff({}, {"a": {"b": 1}}) == {"a": {"b": 1}}
def test_apply_bare_dict_replaces_non_dict():
# Our own producer emits no $replace; a dict over a non-dict old value
# is a wholesale replacement.
assert apply_diff({"a": [1, 2]}, {"a": {"x": 1}}) == {"a": {"x": 1}}
assert apply_diff({"a": 5}, {"a": {"x": 1}}) == {"a": {"x": 1}}
assert apply_diff({"a": None}, {"a": {"x": 1}}) == {"a": {"x": 1}}
def test_apply_list_patch_still_works_on_lists():
# jsondiff-style per-index diff keeps list-op semantics on list state.
assert apply_diff({"l": [1, 2]}, {"l": {"1": 9}}) == {"l": [1, 9]}
# --- jsondiff compatibility, both directions --------------------------------
COMPAT_CASES = [
("scalar change", {"a": 1}, {"a": 2}),
("key add", {"a": 1}, {"a": 1, "b": 2}),
("key remove", {"a": 1, "b": 2}, {"a": 1}),
("last key removed", {"a": 1}, {}),
("nested delete", {"a": {"x": 1, "y": 2}}, {"a": {"x": 1}}),
("nested mixed", {"a": {"x": 1, "y": 2}}, {"a": {"x": 9, "z": 3}}),
("list append", {"l": [1, 2]}, {"l": [1, 2, 3]}),
("list insert mid", {"l": [1, 2, 3]}, {"l": [1, 9, 2, 3]}),
("list remove mid", {"l": [1, 2, 3]}, {"l": [1, 3]}),
("list remove many", {"l": [0, 1, 2, 3, 4]}, {"l": [1, 3]}),
("list replace all", {"l": [1, 2]}, {"l": [3, 4]}),
("list insert+delete", {"l": [0, 1, 2, 3]}, {"l": [9, 1, 8, 3, 7]}),
("dict in list", {"l": [{"x": 1}, {"y": 2}]}, {"l": [{"x": 1}, {"y": 3}]}),
("list to empty", {"l": [1]}, {"l": []}),
("type change dict->list", {"a": {"x": 1}}, {"a": [1]}),
("type change list->dict", {"a": [1]}, {"a": {"x": 1}}),
("dollar key", {"$k": 1, "b": 1}, {"$k": 2}),
("dollar value", {"s": "$x"}, {"s": "$y"}),
(
"deep nesting",
{"a": {"b": {"c": {"d": 1, "e": 2}}}},
{"a": {"b": {"c": {"d": 9}}}},
),
]
# jsondiff.patch cannot apply our patches for "type change list->dict"
# (we emit a bare dict where jsondiff needs $replace) and "dollar value"
# (we do not escape "$"-prefixed values; jsondiff.patch would strip the
# "$"), so those cases are excluded from this direction.
JSONDIFF_APPLIES_CASES = [
c for c in COMPAT_CASES if c[0] not in {"type change list->dict", "dollar value"}
]
@pytest.mark.parametrize(
"name,old,new", JSONDIFF_APPLIES_CASES, ids=[c[0] for c in JSONDIFF_APPLIES_CASES]
)
def test_jsondiff_applies_our_patches(name, old, new):
delta = diff(old, new)
assert delta is not None
assert jsondiff.patch(old, delta, marshal=True) == new
@pytest.mark.parametrize("name,old,new", COMPAT_CASES, ids=[c[0] for c in COMPAT_CASES])
def test_we_apply_jsondiff_patches(name, old, new):
diff = jsondiff.diff(old, new, marshal=True)
assert apply_diff(old, diff) == new
@pytest.mark.parametrize("name,old,new", COMPAT_CASES, ids=[c[0] for c in COMPAT_CASES])
def test_our_own_round_trip(name, old, new):
delta = diff(old, new)
assert delta is not None
assert apply_diff(old, delta) == new
def test_no_diff_means_equal_states():
for _name, old, new in COMPAT_CASES:
assert diff(old, new) is not None # cases really differ
assert diff({"a": [1, {"b": "$x"}]}, {"a": [1, {"b": "$x"}]}) is None
# --- Logging ----------------------------------------------------------------
def test_format_diff_list_edit_shows_whole_list():
diff = jsondiff.diff({"l": [1, 2, 3]}, {"l": [1, 9, 3]}, marshal=True)
lines = format_diff(diff, previous={"l": [1, 2, 3]})
text = "\n".join(lines)
assert "$insert" not in text
assert "[1, 9, 3]" in text
def test_format_diff_unescapes_dollar_keys():
lines = format_diff({"$$weird": 1}, previous={})
assert any("$weird" in line and "$$weird" not in line for line in lines)
+55
View File
@@ -0,0 +1,55 @@
"""Tests for LockedFile low-level behaviors."""
from kanta.exceptions import FileLockError
from kanta.filelock import LockedFile
def test_replace_content_rewrites_in_place(tmp_path):
path = tmp_path / "data.kantadb"
path.write_bytes(b"original content here")
f = LockedFile()
f.open(path)
try:
f.replace_content(b"new")
assert f.size() == 3
f.write(b"!")
finally:
f.close()
assert path.read_bytes() == b"new!"
def test_replace_content_keeps_lock(tmp_path):
path = tmp_path / "data.kantadb"
path.write_bytes(b"abc")
f = LockedFile()
f.open(path)
try:
f.replace_content(b"xyz")
other = LockedFile()
try:
other.open(path)
raise AssertionError("second open should fail while lock is held")
except FileLockError:
pass
finally:
f.close()
def test_replace_content_grow_and_shrink(tmp_path):
path = tmp_path / "data.kantadb"
path.write_bytes(b"x" * 100)
f = LockedFile()
f.open(path)
try:
f.replace_content(b"")
assert f.size() == 0
f.replace_content(b"y" * 200)
assert f.size() == 200
finally:
f.close()
assert path.read_bytes() == b"y" * 200
+26
View File
@@ -1,4 +1,8 @@
from kanta.logging import format_diff from kanta.logging import format_diff
from kanta.tty import ESC, colors
_ADD = f"{ESC}{colors.add}m"
_DELETE = f"{ESC}{colors.delete}m"
def test_add(): def test_add():
@@ -6,6 +10,28 @@ def test_add():
assert any("name" in line for line in lines) assert any("name" in line for line in lines)
def test_add_path_is_green():
lines = format_diff({"name": "Alice"}, previous={})
assert any(_ADD in line for line in lines)
def test_nested_add_path_final_element_is_green():
lines = format_diff({"users": {"alice": 1}}, previous={"users": {}})
assert any(_ADD in line and "alice" in line for line in lines)
def test_update_path_not_colored_as_add():
lines = format_diff({"name": "Bob"}, previous={"name": "Alice"})
assert lines
assert all(_ADD not in line for line in lines)
def test_delete_path_not_colored_as_add():
lines = format_diff({"$delete": ["old_key"]}, previous={"old_key": 1})
assert any(_DELETE in line for line in lines)
assert all(_ADD not in line for line in lines)
def test_update(): def test_update():
lines = format_diff({"name": "Bob"}, previous={"name": "Alice"}) lines = format_diff({"name": "Bob"}, previous={"name": "Alice"})
assert any("Bob" in line for line in lines) assert any("Bob" in line for line in lines)
+689
View File
@@ -0,0 +1,689 @@
"""Tests for structural ``--grep`` matching and match highlighting."""
from datetime import UTC, datetime
from kanta.__main__ import main
from kanta.grep import (
GrepPattern,
_Entry,
_find_spans,
_match_path,
_match_value,
evaluate,
mark_spans,
matches_snapshot,
)
from kanta.logging import _USER_PATH
from kanta.serialization import JsonSerializer
from kanta.serialization.framing import LineFramer
from kanta.structs import ChangeRecord, Snapshot
from kanta.tty import strip_ansi
TS = datetime(2026, 1, 1, tzinfo=UTC)
MARK = "\x1b[48;5;220m"
UNMARK = "\x1b[49m"
def test_match_path_whole_elements_anywhere():
assert _match_path("users", ["users"]) == frozenset({0})
assert _match_path("users", ["data", "users", "alice"]) == frozenset({1})
# Partial element matches require explicit wildcards.
assert _match_path("users", ["foousers"]) is None
assert _match_path("us*rs", ["foousers"]) is None
assert _match_path("*users", ["foousers"]) == frozenset({0})
def test_match_path_contiguous_element_sequence():
assert _match_path("alice.email", ["users", "alice", "email"]) == frozenset({1, 2})
# The sequence must be contiguous.
assert _match_path("users.email", ["users", "alice", "email"]) is None
assert _match_path("users.*.email", ["users", "alice", "email"]) == frozenset(
{0, 1, 2}
)
# Matching is case-insensitive and empty terms match without marking.
assert _match_path("USERS", ["users"]) == frozenset({0})
assert _match_path("", ["anything"]) == frozenset()
def test_match_value_substring_for_strings_full_for_scalars():
assert _match_value("lice", "alice@example.com", "alice@example.com")
assert not _match_value("bob", "alice@example.com", "alice@example.com")
# Booleans, numbers and null match only in full.
assert _match_value("true", True, "true")
assert not _match_value("tru", True, "true")
assert _match_value("42", 42, "42")
assert not _match_value("4", 42, "42")
assert _match_value("null", None, "null")
# Wildcards match the whole value text of any type.
assert _match_value("tru*", True, "true")
assert _match_value("4*", 42, "42")
# Containers do not match; their leaves are matched individually.
assert not _match_value("email", {"email": "a@b.c"}, '{"email": "a@b.c"}')
# An empty term matches anything.
assert _match_value("", True, "true")
def test_mark_spans_merges_overlaps_and_adjacents():
assert mark_spans("alice@example.com", [(0, 5), (3, 11)]) == (
f"{MARK}alice@examp{UNMARK}le.com"
)
assert mark_spans("aab", [(0, 1), (1, 3)]) == f"{MARK}aab{UNMARK}"
assert mark_spans("ab", [(1, 1), (2, 2)]) == "ab"
def test_mark_spans_offsets_into_styled_text():
styled = "\x1b[32mal\x1b[0mlice"
# A span crossing an escape sequence wraps each escape-free run.
assert mark_spans(styled, [(0, 3)]) == (
f"\x1b[32m{MARK}al{UNMARK}\x1b[0m{MARK}l{UNMARK}ice"
)
# A span covering everything wraps each escape-free run separately.
assert mark_spans(styled, [(0, 6)]) == (
f"\x1b[32m{MARK}al{UNMARK}\x1b[0m{MARK}lice{UNMARK}"
)
assert mark_spans(styled, []) == styled
def test_find_spans_case_insensitive_occurrences():
assert _find_spans("Alice likes ALICE", "alice") == [(0, 5), (12, 17)]
assert _find_spans("nope", "alice") == []
assert _find_spans("anything", "") == []
def test_pattern_parse_forms():
bare = GrepPattern.parse("alice")
assert bare.term == "alice" and bare.path_term is None
pair = GrepPattern.parse("users.alice.age=30")
assert pair.term is None
assert pair.path_term == "users.alice.age"
assert pair.value_term == "30"
value_only = GrepPattern.parse("=alice@example.com")
assert value_only.path_term == "" and value_only.value_term == "alice@example.com"
path_only = GrepPattern.parse("users.alice=")
assert path_only.path_term == "users.alice" and path_only.value_term == ""
# Split on the first '=' only; the value may contain '='.
multi = GrepPattern.parse("key=a=b")
assert multi.path_term == "key" and multi.value_term == "a=b"
def _patterns(*raws: str) -> list[GrepPattern]:
return [GrepPattern.parse(raw) for raw in raws]
def test_evaluate_bare_term_against_path_value_action_user():
record = ChangeRecord(
ts=TS, a="create_user", u="admin", diff={"users": {"alice": {"age": 30}}}
)
assert evaluate(record, {}, _patterns("users.alice"))
assert evaluate(record, {}, _patterns("30"))
assert evaluate(record, {}, _patterns("CREATE_user"))
assert evaluate(record, {}, _patterns("admin"))
assert evaluate(record, {}, _patterns("age=30"))
assert not evaluate(record, {}, _patterns("bob"))
assert not evaluate(record, {}, _patterns("3")) # not a full number match
def test_evaluate_unescapes_dollar_keys():
record = ChangeRecord(ts=TS, a="set", diff={"$$config": 5})
assert evaluate(record, {}, _patterns("$config=5"))
def test_evaluate_path_value_form_requires_same_line():
record = ChangeRecord(ts=TS, a="set", diff={"a": {"x": 1}, "b": {"y": 2}})
previous = {"a": {"x": 0}, "b": {"y": 0}}
assert evaluate(record, previous, _patterns("a.x=1"))
# 'a' matches one line's path, '2' another line's value: no match.
assert not evaluate(record, previous, _patterns("a=2"))
def test_evaluate_all_patterns_same_record_any_line():
record = ChangeRecord(ts=TS, a="set", diff={"a": {"x": 1}, "b": {"y": 2}})
previous = {"a": {"x": 0}, "b": {"y": 0}}
assert evaluate(record, previous, _patterns("a.x", "=2"))
assert not evaluate(record, previous, _patterns("a.x", "=2", "missing"))
def test_evaluate_matches_deleted_content_by_previous_value():
record = ChangeRecord(ts=TS, a="delete_user", diff={"users": {"$delete": "alice"}})
previous = {"users": {"alice": {"email": "alice@example.com"}}}
assert evaluate(record, previous, _patterns("alice@example.com"))
assert evaluate(record, previous, _patterns("users.alice.email"))
assert not evaluate(record, {}, _patterns("alice@example.com"))
def test_highlighter_marks_exactly_the_matched_regions():
record = ChangeRecord(
ts=TS,
a="set",
diff={
"users": {"alice": {"email": "same@x.com"}, "bob": {"email": "same@x.com"}}
},
)
previous = {
"users": {"alice": {"email": "old@x.com"}, "bob": {"email": "same@x.com"}}
}
hl = evaluate(record, previous, _patterns("users.alice.email"))
assert hl is not None
# Path-side match: the matched elements are lit, the value is not.
assert hl.path("alice", "users.alice") == f"{MARK}alice{UNMARK}"
assert hl.value("same@x.com", "users.alice.email") == "same@x.com"
# Bob's identical value is a different path: untouched.
assert hl.path("bob", "users.bob") == "bob"
assert hl.value("same@x.com", "users.bob.email") == "same@x.com"
hl = evaluate(record, previous, _patterns("=same@x.com"))
assert hl is not None
# Both lines genuinely match the value: both are marked.
assert hl.value("same@x.com", "users.alice.email") == f"{MARK}same@x.com{UNMARK}"
assert hl.value("same@x.com", "users.bob.email") == f"{MARK}same@x.com{UNMARK}"
assert hl.path("alice", "users.alice") == "alice"
def test_highlighter_merges_overlapping_needles_from_different_patterns():
record = ChangeRecord(ts=TS, a="set", diff={"email": "alice@example.com"})
hl = evaluate(record, {}, _patterns("alice", "lice@exam"))
assert hl is not None
assert hl.value("alice@example.com", "email") == (
f"{MARK}alice@exam{UNMARK}ple.com"
)
def test_highlighter_marks_delete_marker_on_removed_content_match():
record = ChangeRecord(ts=TS, a="delete_user", diff={"users": {"$delete": "bob"}})
previous = {"users": {"bob": {"email": "bob@example.com"}}}
hl = evaluate(record, previous, _patterns("bob@example.com"))
assert hl is not None
# The matched content is gone: the deletion marker is lit, not the path.
assert hl.path("users", "users") == "users"
assert hl.path("bob", "users.bob") == "bob"
assert hl.delete("", "users.bob") == f"{MARK}{UNMARK}"
assert hl.delete("", "users.alice") == ""
# A path-side match lights the genuinely matched elements of the anchor,
# and the marker for the element below it.
hl = evaluate(record, previous, _patterns("users.bob.email"))
assert hl is not None
assert hl.path("users", "users") == f"{MARK}users{UNMARK}"
assert hl.path("bob", "users.bob") == f"{MARK}bob{UNMARK}"
assert hl.delete("", "users.bob") == f"{MARK}{UNMARK}"
# A path-side match of one displayed element lights only that element.
hl = evaluate(record, previous, _patterns("bob"))
assert hl is not None
assert hl.path("users", "users") == "users"
assert hl.path("bob", "users.bob") == f"{MARK}bob{UNMARK}"
def test_highlighter_meta_marks_action_and_user():
record = ChangeRecord(ts=TS, a="create_user", u="admin", diff={"x": 1})
hl = evaluate(record, {}, _patterns("create", "ADM"))
assert hl is not None
assert hl.meta("create_user", "action") == f"{MARK}create{UNMARK}_user"
assert hl.meta("admin", "user") == f"{MARK}adm{UNMARK}in"
assert hl.meta("extra", "other") == "extra"
def test_matches_snapshot_uses_change_record_matching():
state = {"users": {"alice": {"email": "alice@example.com", "age": 30}}}
assert matches_snapshot(state, _patterns("users.alice"))
assert matches_snapshot(state, _patterns("alice@example"))
assert matches_snapshot(state, _patterns("age=30"))
assert matches_snapshot(state, _patterns("users.alice", "=30"))
assert not matches_snapshot(state, _patterns("bob"))
assert not matches_snapshot(state, _patterns("3")) # not a full number match
assert not matches_snapshot(state, _patterns("alice", "missing"))
assert not matches_snapshot({}, _patterns("alice"))
def test_matches_snapshot_supports_logfmt_prettified_values():
state = {"when": 1767225600}
def logfmt(value, path):
return "2026-01-01" if value == 1767225600 else None
assert matches_snapshot(state, _patterns("2026"), logfmt=logfmt)
assert matches_snapshot(state, _patterns("1767225600"), logfmt=logfmt)
assert not matches_snapshot(state, _patterns("2026"))
def test_entry_dataclass_holds_anchor_for_deletes():
entry = _Entry(["a", "b"], 1, "1", anchor=["a"])
assert entry.anchor == ["a"]
def _write_db(path, changes, state=None):
serializer = JsonSerializer()
framer = LineFramer()
snapshot = Snapshot(ts=TS, v=1, state=state or {})
data = framer.frame_snapshot(serializer.encode(snapshot), record_offset=0)
for change in changes:
data += framer.frame_change(serializer.encode(change), record_offset=0)
path.write_bytes(data)
def _sample_changes():
return [
ChangeRecord(
ts=TS,
a="create_alice",
u="admin",
diff={
"users": {
"alice": {"email": "alice@example.com", "admin": True},
}
},
),
ChangeRecord(
ts=TS,
a="update_alice",
u="admin",
diff={"users": {"alice": {"age": 30}}},
),
ChangeRecord(
ts=TS,
a="create_bob",
u="bob",
diff={"users": {"bob": {"email": "bob@example.com"}}},
),
]
def _run_cli(tmp_path, capsys, monkeypatch, changes, *args, color=False, state=None):
if color:
monkeypatch.setenv("FORCE_COLOR", "1")
monkeypatch.delenv("NO_COLOR", raising=False)
else:
monkeypatch.setenv("NO_COLOR", "1")
monkeypatch.delenv("FORCE_COLOR", raising=False)
path = tmp_path / "test.kantadb"
_write_db(path, changes, state=state)
code = main([str(path), *args])
assert code == 0
return capsys.readouterr().err
def test_cli_grep_filters_records(tmp_path, capsys, monkeypatch):
"""Only matching change records are printed."""
err = _run_cli(tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "alice")
assert "create_alice" in err
assert "update_alice" in err
assert "create_bob" not in err
assert "bob@example.com" not in err
def test_cli_grep_prints_entire_transaction(tmp_path, capsys, monkeypatch):
"""A match prints the whole record, not just the matching line."""
err = _run_cli(
tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "users.alice.email"
)
assert "create_alice" in err
# Non-matching lines of the same record are printed too.
assert "admin" in err and "true" in err
# The update record does not contain the path.
assert "update_alice" not in err
def test_cli_grep_repeated_patterns_must_all_match(tmp_path, capsys, monkeypatch):
"""Repeated --grep options are ANDed within the same record."""
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_sample_changes(),
"--grep",
"alice",
"--grep",
"=alice@example.com",
)
assert "create_alice" in err
# update_alice matches 'alice' but has no email value.
assert "update_alice" not in err
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_sample_changes(),
"--grep",
"alice",
"--grep",
"bob",
)
assert "create_alice" not in err
assert "create_bob" not in err
def test_cli_grep_no_match_exits_zero(tmp_path, capsys, monkeypatch):
"""No matching records is not an error; non-matching snapshots are hidden."""
err = _run_cli(tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "nobody")
assert "snapshot s0" not in err
assert "create_alice" not in err
assert "create_bob" not in err
def test_cli_grep_snapshot_prints_only_when_state_matches(
tmp_path, capsys, monkeypatch
):
"""Snapshots are matched against their full state like change records."""
state = {"users": {"alice": {"email": "alice@example.com", "age": 30}}}
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_sample_changes(),
"--grep",
"alice",
state=state,
)
assert "snapshot s0" in err
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_sample_changes(),
"--grep",
"users.alice.age=30",
state=state,
)
assert "snapshot s0" in err
# A pattern matching nothing in the state suppresses the snapshot.
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_sample_changes(),
"--grep",
"bob",
state=state,
)
assert "snapshot s0" not in err
# Repeated patterns are ANDed within the snapshot state too.
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_sample_changes(),
"--grep",
"alice",
"--grep",
"missing",
state=state,
)
assert "snapshot s0" not in err
def test_cli_grep_path_value_forms(tmp_path, capsys, monkeypatch):
"""The 'path=value', 'path=' and '=value' forms restrict the match side."""
err = _run_cli(
tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "users.*.admin=true"
)
assert "create_alice" in err
assert "create_bob" not in err
err = _run_cli(
tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "users.alice.age="
)
assert "update_alice" in err
assert "create_alice" not in err
err = _run_cli(
tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "=bob@example.com"
)
assert "create_bob" in err
assert "create_alice" not in err
def test_cli_grep_path_elements_match_in_full(tmp_path, capsys, monkeypatch):
"""'users' does not match a 'foousers' element; 'us*' does."""
changes = [ChangeRecord(ts=TS, a="trap", diff={"foousers": {"note": "x"}})]
err = _run_cli(tmp_path, capsys, monkeypatch, changes, "--grep", "users")
assert "trap" not in err
err = _run_cli(tmp_path, capsys, monkeypatch, changes, "--grep", "*users")
assert "trap" in err
def _shared_email_changes():
return [
ChangeRecord(
ts=TS,
a="create_alice",
u="admin",
diff={"users": {"alice": {"email": "shared@example.com", "admin": True}}},
),
ChangeRecord(
ts=TS,
a="create_bob",
u="admin",
diff={"users": {"bob": {"email": "shared@example.com"}}},
),
ChangeRecord(
ts=TS,
a="delete_bob",
u="admin",
diff={"users": {"$delete": "bob"}},
),
]
def _lines_with(err: str, text: str) -> list[str]:
return [line for line in err.splitlines() if text in strip_ansi(line)]
def test_cli_highlight_marks_exactly_the_matched_regions(tmp_path, capsys, monkeypatch):
"""A path-side match lights only the matched elements, not the value."""
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_shared_email_changes(),
"--grep",
"users.alice.email",
color=True,
)
(alice_line,) = _lines_with(err, "shared@example.com")
assert f"{MARK}alice{UNMARK}" in alice_line
assert f"{MARK}email{UNMARK}" in alice_line
# The value itself did not match, so it is not highlighted.
assert f"{MARK}shared@example.com{UNMARK}" not in alice_line
(header_line,) = _lines_with(err, "users =")
assert f"{MARK}users{UNMARK}" in header_line
assert "create_bob" not in err
def test_cli_highlight_value_matches_on_all_matching_lines(
tmp_path, capsys, monkeypatch
):
"""A value-side match lights the value on every line that genuinely matched."""
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_shared_email_changes(),
"--grep",
"=shared@example.com",
color=True,
)
lines = _lines_with(err, "shared@example.com")
assert len(lines) == 2 # alice's and bob's records both matched
for line in lines:
assert f"{MARK}shared@example.com{UNMARK}" in line
# The deletion of bob matched by its previous (removed) value: the
# deletion marker is lit, not the deleted path.
(delete_line,) = _lines_with(err, "")
assert f"{MARK}{UNMARK}" in delete_line
assert f"{MARK}bob{UNMARK}" not in delete_line
assert f"{MARK}users{UNMARK}" not in delete_line
def test_cli_highlight_merges_overlapping_matches(tmp_path, capsys, monkeypatch):
"""Overlapping matches from different patterns form one continuous mark."""
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_shared_email_changes(),
"--grep",
"shared",
"--grep",
"red@exam",
color=True,
)
lines = _lines_with(err, "shared@example.com")
assert len(lines) == 2 # both records genuinely match both patterns
for line in lines:
assert f"{MARK}shared@exam{UNMARK}ple.com" in line
def test_cli_highlight_full_scalar_and_header_fields(tmp_path, capsys, monkeypatch):
"""Scalars are marked in full; matched action/user substrings are marked."""
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
_shared_email_changes(),
"--grep",
"admin=true",
"--grep",
"create_al",
color=True,
)
(line,) = _lines_with(err, "shared@example.com")
assert f"{MARK}true{UNMARK}" in line
(header,) = _lines_with(err, "create_alice")
assert f"{MARK}create_al{UNMARK}ice" in header
# Only the record matching both patterns is printed.
assert "create_bob" not in err
assert "delete_bob" not in err
def test_evaluate_matches_prettified_and_raw_value_forms():
"""With logfmt, both the raw and the prettified form of a value match."""
record = ChangeRecord(ts=TS, a="set", diff={"when": 1767225600})
def logfmt(value, path):
return "2026-01-01" if value == 1767225600 else None
# The prettified form matches; the needle is located in the display text.
hl = evaluate(record, {}, _patterns("2026"), logfmt=logfmt)
assert hl is not None
assert hl.value("2026-01-01", "when") == f"{MARK}2026{UNMARK}-01-01"
# The raw form matches too; its needle is absent from the displayed
# text, so the whole displayed value is marked.
hl = evaluate(record, {}, _patterns("1767225600"), logfmt=logfmt)
assert hl is not None
assert hl.value("2026-01-01", "when") == f"{MARK}2026-01-01{UNMARK}"
# Without logfmt the raw value is matched and marked precisely.
hl = evaluate(record, {}, _patterns("1767225600"))
assert hl.value("1767225600", "when") == f"{MARK}1767225600{UNMARK}"
# Neither form matches.
assert evaluate(record, {}, _patterns("1999"), logfmt=logfmt) is None
def test_evaluate_matches_prettified_user():
"""The user field matches both the raw id and the logfmt-resolved name."""
def logfmt(value, path):
return "Alice Admin" if path == _USER_PATH else None
record = ChangeRecord(ts=TS, a="set", u="u123", diff={"x": 1})
hl = evaluate(record, {}, _patterns("alice"), logfmt=logfmt)
assert hl is not None
assert hl.meta("Alice Admin", "user") == f"{MARK}Alice{UNMARK} Admin"
# A raw-only user match marks the whole displayed name.
hl = evaluate(record, {}, _patterns("u123"), logfmt=logfmt)
assert hl is not None
assert hl.meta("Alice Admin", "user") == f"{MARK}Alice Admin{UNMARK}"
# The action is never prettified.
assert hl.meta("set", "action") == "set"
def test_evaluate_without_logfmt_matches_raw_only():
record = ChangeRecord(ts=TS, a="set", u="u123", diff={"when": 1767225600})
assert evaluate(record, {}, _patterns("1767225600"))
assert evaluate(record, {}, _patterns("2026")) is None
assert evaluate(record, {}, _patterns("alice")) is None
def test_cli_grep_matches_prettified_forms_with_kanta_object(
tmp_path, capsys, monkeypatch
):
"""End to end: a -k object's logfmt formatter doubles the match surface."""
db_path = tmp_path / "test.kantadb"
module = tmp_path / "dbmod.py"
module.write_text(
"from typing import Any\n"
"from kanta import Kanta\n"
f"kanta = Kanta({str(db_path)!r}, {{}}, type=dict)\n"
"@kanta.logfmt\n"
"def pretty(value: Any, path: str) -> str | None:\n"
" if path == 'when':\n"
" return 'Nov 3, 2025'\n"
" if path == '$user':\n"
" return 'Alice Admin'\n"
" return None\n"
)
changes = [ChangeRecord(ts=TS, a="set", u="u123", diff={"when": 1767225600})]
kanta_args = ("-k", str(module))
# A term matching only the prettified value finds the record.
err = _run_cli(
tmp_path, capsys, monkeypatch, changes, *kanta_args, "--grep", "nov", color=True
)
(line,) = _lines_with(err, "Nov 3, 2025")
assert f"{MARK}Nov{UNMARK} 3, 2025" in line
# A term matching only the raw value marks the whole prettified display.
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
changes,
*kanta_args,
"--grep",
"1767225600",
color=True,
)
(line,) = _lines_with(err, "Nov 3, 2025")
assert f"{MARK}Nov 3, 2025{UNMARK}" in line
# The prettified user matches, and the raw id marks the whole display.
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
changes,
*kanta_args,
"--grep",
"alice",
color=True,
)
(header,) = _lines_with(err, "Alice Admin")
assert f"{MARK}Alice{UNMARK} Admin" in header
err = _run_cli(
tmp_path,
capsys,
monkeypatch,
changes,
*kanta_args,
"--grep",
"u123",
color=True,
)
(header,) = _lines_with(err, "Alice Admin")
assert f"{MARK}Alice Admin{UNMARK}" in header
# Without -k there is no prettified form to match.
err = _run_cli(tmp_path, capsys, monkeypatch, changes, "--grep", "nov")
assert not _lines_with(err, "u123")
err = _run_cli(tmp_path, capsys, monkeypatch, changes, "--grep", "1767225600")
assert _lines_with(err, "u123")
+381 -2
View File
@@ -1,4 +1,5 @@
import asyncio import asyncio
import logging
import sys import sys
from datetime import UTC, datetime from datetime import UTC, datetime
from uuid import uuid4 from uuid import uuid4
@@ -6,6 +7,7 @@ from uuid import uuid4
import pytest import pytest
from kanta.exceptions import DatabaseError, DataIntegrityError, FileLockError from kanta.exceptions import DatabaseError, DataIntegrityError, FileLockError
from kanta.migrations import MigrationResult
from kanta.serialization import struct_to_dict from kanta.serialization import struct_to_dict
from .support import ( from .support import (
@@ -17,6 +19,9 @@ from .support import (
change_actions, change_actions,
fixed_change, fixed_change,
make_kanta, make_kanta,
make_migrations_module,
read_changes,
read_last_snapshot,
seed_single_change, seed_single_change,
) )
@@ -30,6 +35,64 @@ async def test_load_empty(tmp_path, format_config):
await kanta.close() await kanta.close()
@pytest.mark.asyncio
async def test_new_file_writes_bootstrap_record_without_handlers(
tmp_path, format_config
):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
await kanta.open()
await kanta.close()
records = read_changes(path, format_config)
assert len(records) == 1
assert records[0].a == "bootstrap"
assert records[0].diff == {"users": {}, "counter": 0}
@pytest.mark.asyncio
async def test_new_file_persists_initial_state_for_roundtrip(tmp_path, format_config):
path = tmp_path / "test.db"
kanta = make_kanta(
path, Data(counter=5, users={"alice": User(name="Alice")}), format_config
)
await kanta.open()
await kanta.close()
records = read_changes(path, format_config)
assert len(records) == 1
assert records[0].a == "bootstrap"
assert records[0].diff == {
"users": {"alice": {"name": "Alice", "age": 0}},
"counter": 5,
}
kanta2 = make_kanta(path, Data, format_config)
await kanta2.open()
assert kanta2.data.counter == 5
assert kanta2.data.users["alice"].name == "Alice"
await kanta2.close()
@pytest.mark.asyncio
async def test_reopen_without_changes_does_not_force_snapshot(tmp_path, format_config):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data(counter=5), format_config)
await kanta.open()
await kanta.close()
# No snapshot should exist after the initial bootstrap and close.
assert read_last_snapshot(path, format_config) is None
kanta2 = make_kanta(path, Data, format_config)
await kanta2.open()
assert kanta2.data.counter == 5
await kanta2.close()
# Re-opening without migrations or normalization changes must not force one.
assert read_last_snapshot(path, format_config) is None
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_open_overwrites_caller_owned_root_data(tmp_path, format_config): async def test_open_overwrites_caller_owned_root_data(tmp_path, format_config):
path = tmp_path / "test.db" path = tmp_path / "test.db"
@@ -404,7 +467,7 @@ async def test_migrations_from_module(tmp_path, format_config):
mod = type(sys)("test_migrations") mod = type(sys)("test_migrations")
def migrate_v1(d, ctx): def migrate_v1(d, kanta):
d["version"] = 1 d["version"] = 1
mod.__dict__["migrate_v1"] = migrate_v1 mod.__dict__["migrate_v1"] = migrate_v1
@@ -434,6 +497,322 @@ async def test_msgspec_normalization_logs_migration(tmp_path, format_config):
assert "migrate:msgspec" in change_actions(path, format_config) assert "migrate:msgspec" in change_actions(path, format_config)
@pytest.mark.asyncio
async def test_empty_migration_writes_snapshot_and_is_not_reapplied(
tmp_path, format_config
):
path = tmp_path / "test.db"
seed_single_change(
path, fixed_change("init", {"counter": 0, "users": {}}), format_config
)
def migrate_v1(d, kanta):
"""No-op migration that only bumps the schema version."""
pass
mod = make_migrations_module("empty_migration_mod", "migrate_v1", migrate_v1)
try:
kanta = make_kanta(path, Data, format_config, migrations=mod)
await kanta.open()
assert kanta.version == 1
await kanta.close()
# Empty migrations must not produce empty change records.
records = read_changes(path, format_config)
migration_records = [r for r in records if r.a.startswith("migrate")]
assert not migration_records
# The version bump is persisted via a snapshot instead.
snap = read_last_snapshot(path, format_config)
assert snap is not None
assert snap.v == 1
assert snap.state == {"counter": 0, "users": {}}
kanta2 = make_kanta(path, Data, format_config, migrations=mod)
await kanta2.open()
assert kanta2.version == 1
await kanta2.close()
# Re-opening must not create additional migration records or snapshots.
records2 = read_changes(path, format_config)
assert not [r for r in records2 if r.a.startswith("migrate")]
finally:
sys.modules.pop("empty_migration_mod", None)
@pytest.mark.asyncio
async def test_migration_with_changes_records_diff_and_snapshot(
tmp_path, format_config
):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("init", {"counter": 0}), format_config)
mod = type(sys)("test_migrations_changes")
def migrate_v1(d, kanta):
d["counter"] = 2
mod.__dict__["migrate_v1"] = migrate_v1
kanta = make_kanta(path, Data, format_config, migrations=mod)
await kanta.open()
assert kanta.version == 1
assert kanta.data.counter == 2
await kanta.close()
records = read_changes(path, format_config)
migration_records = [r for r in records if r.a.startswith("migrate")]
# The version migration and the msgspec normalization that follows it are
# grouped into a single migrate:vN record.
assert len(migration_records) == 1
assert migration_records[0].a == "migrate:v1"
assert migration_records[0].v == 1
assert migration_records[0].diff == {"counter": 2, "users": {}}
snap = read_last_snapshot(path, format_config)
assert snap is not None
assert snap.v == 1
assert snap.state == {"counter": 2, "users": {}}
@pytest.mark.asyncio
async def test_migration_summary_log_includes_filename(tmp_path, format_config, caplog):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("init", {"counter": 0}), format_config)
mod = type(sys)("test_migrations_log")
def migrate_v1(d, kanta):
"""Bump counter."""
d["counter"] = 2
mod.__dict__["migrate_v1"] = migrate_v1
with caplog.at_level(logging.INFO, logger="kanta.migration"):
kanta = make_kanta(path, Data, format_config, migrations=mod)
await kanta.open()
assert kanta.version == 1
await kanta.close()
info_messages = [r.message for r in caplog.records if r.levelno == logging.INFO]
assert len(info_messages) == 1
assert str(path) in info_messages[0]
assert "v0 -> v1" in info_messages[0]
assert "migrate_v1 (Bump counter)" in info_messages[0]
@pytest.mark.asyncio
async def test_open_log_false_suppresses_migration_log(tmp_path, format_config, caplog):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("init", {"counter": 0}), format_config)
mod = type(sys)("test_migrations_silent")
def migrate_v1(d, kanta):
d["counter"] = 2
mod.__dict__["migrate_v1"] = migrate_v1
with caplog.at_level(logging.INFO, logger="kanta.migration"):
kanta = make_kanta(path, Data, format_config, migrations=mod)
await kanta.open(log=False)
await kanta.close()
info_messages = [r for r in caplog.records if r.levelno == logging.INFO]
assert not info_messages
@pytest.mark.asyncio
async def test_open_log_true_logs_bootstrap(tmp_path, format_config, caplog):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
with caplog.at_level(logging.INFO, logger="kanta.bootstrap"):
await kanta.open()
await kanta.close()
info_messages = [r.message for r in caplog.records if r.levelno == logging.INFO]
assert len(info_messages) >= 2
assert "created" in info_messages[0]
assert "bootstrap" in info_messages[1]
@pytest.mark.asyncio
async def test_open_log_false_suppresses_bootstrap_log(tmp_path, format_config, caplog):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
with caplog.at_level(logging.INFO, logger="kanta.bootstrap"):
await kanta.open(log=False)
await kanta.close()
info_messages = [r for r in caplog.records if r.levelno == logging.INFO]
assert not info_messages
@pytest.mark.asyncio
async def test_open_log_custom_logger_logs_bootstrap(tmp_path, format_config, caplog):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
custom_logger = logging.getLogger("custom.bootstrap")
custom_logger.setLevel(logging.INFO)
with caplog.at_level(logging.INFO, logger="custom.bootstrap"):
await kanta.open(log=custom_logger)
await kanta.close()
info_messages = [r.message for r in caplog.records if r.levelno == logging.INFO]
assert len(info_messages) >= 2
assert "created" in info_messages[0]
assert "bootstrap" in info_messages[1]
@pytest.mark.asyncio
async def test_open_existing_database_logs_using_on_debug(
tmp_path, format_config, caplog
):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
await kanta.open()
await kanta.close()
kanta2 = make_kanta(path, Data, format_config)
with caplog.at_level(logging.DEBUG, logger="kanta.bootstrap"):
await kanta2.open()
await kanta2.close()
debug_messages = [r.message for r in caplog.records if r.levelno == logging.DEBUG]
assert any("opened" in m and str(path.resolve()) in m for m in debug_messages)
@pytest.mark.asyncio
async def test_logmigr_callback_replaces_default_logging(
tmp_path, format_config, caplog
):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("init", {"counter": 0}), format_config)
mod = type(sys)("test_migrations_callback")
def migrate_v1(d, kanta):
"""Bump counter."""
d["counter"] = 2
mod.__dict__["migrate_v1"] = migrate_v1
summaries = []
kanta = make_kanta(path, Data, format_config, migrations=mod)
@kanta.logmigr
def collect(summary: MigrationResult):
summaries.append(summary)
with caplog.at_level(logging.INFO, logger="kanta.migration"):
await kanta.open()
await kanta.close()
assert len(summaries) == 1
assert summaries[0].version == 1
assert summaries[0].migrations[0].name == "migrate_v1"
info_messages = [r for r in caplog.records if r.levelno == logging.INFO]
assert not info_messages
@pytest.mark.asyncio
async def test_logmigr_callback_report(tmp_path, format_config, caplog):
import logging
from kanta import MigrationReport
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("init", {"counter": 0}), format_config)
mod = type(sys)("test_migrations_report")
def migrate_v1(d):
"""Bump counter."""
d["counter"] = 2
mod.__dict__["migrate_v1"] = migrate_v1
reports = []
kanta = make_kanta(path, Data, format_config, migrations=mod)
@kanta.logmigr
def collect(report: MigrationReport):
reports.append(report)
with caplog.at_level(logging.INFO, logger="kanta.migration"):
await kanta.open()
await kanta.close()
assert len(reports) == 1
assert reports[0].original == 0
assert reports[0].version == 1
assert [m.name for m in reports[0].applied] == ["migrate_v1"]
@pytest.mark.asyncio
async def test_transaction_log_false_suppresses_log(tmp_path, format_config, caplog):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
await kanta.open()
with caplog.at_level(logging.INFO, logger="kanta.transaction"):
with kanta.transaction(action="inc", log=False) as data:
data.counter = 1
await kanta.close()
info_messages = [r for r in caplog.records if r.levelno == logging.INFO]
assert not info_messages
@pytest.mark.asyncio
async def test_transaction_logdiff_false_logs_header_only(
tmp_path, format_config, caplog
):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
await kanta.open()
with caplog.at_level(logging.INFO, logger="kanta.transaction"):
with kanta.transaction(action="inc", logdiff=False) as data:
data.counter = 1
await kanta.close()
messages = [r.message for r in caplog.records if r.levelno == logging.INFO]
assert len(messages) == 1
assert "inc" in messages[0]
assert "counter" not in messages[0]
@pytest.mark.asyncio
async def test_transaction_log_custom_logger(tmp_path, format_config, caplog):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
await kanta.open()
custom_logger = logging.getLogger("custom.transaction")
custom_logger.setLevel(logging.INFO)
with caplog.at_level(logging.INFO, logger="custom.transaction"):
with kanta.transaction(action="inc", log=custom_logger) as data:
data.counter = 1
await kanta.close()
info_messages = [r for r in caplog.records if r.levelno == logging.INFO]
assert len(info_messages) >= 1
assert "inc" in info_messages[0].message
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_open_locked_file_raises_filelock_error(tmp_path, format_config): async def test_open_locked_file_raises_filelock_error(tmp_path, format_config):
path = tmp_path / "test.db" path = tmp_path / "test.db"
@@ -514,7 +893,7 @@ async def test_migrations_from_module_path(tmp_path, format_config):
module_name = "test_migrations_path" module_name = "test_migrations_path"
mod = type(sys)(module_name) mod = type(sys)(module_name)
def migrate_v1(d, ctx): def migrate_v1(d, kanta):
d["counter"] = 2 d["counter"] = 2
mod.__dict__["migrate_v1"] = migrate_v1 mod.__dict__["migrate_v1"] = migrate_v1
+388
View File
@@ -0,0 +1,388 @@
import logging
import sys
import pytest
from kanta.logging import (
LogEvent,
bootstrap_logger,
configure_logging,
emit_event,
log_change,
migration_logger,
transaction_logger,
)
from kanta.migrations import MigrationResult
from tests.support import (
Data,
fixed_change,
make_kanta,
seed_single_change,
)
@pytest.fixture(autouse=True)
def _reset_kanta_loggers():
yield
for name in (
"kanta",
"kanta.transaction",
"kanta.transaction.diff",
"kanta.bootstrap",
"kanta.migration",
):
logger = logging.getLogger(name)
logger.setLevel(logging.NOTSET)
logger.propagate = True
logger.disabled = False
logger.handlers.clear()
def _setup_logging(**kwargs):
"""Default kanta logging with the event loggers lifted to INFO.
Event loggers inherit the root level (WARNING under pytest); output
assertions need INFO.
"""
configure_logging(**kwargs)
for name in ("kanta.bootstrap", "kanta.migration", "kanta.transaction"):
logging.getLogger(name).setLevel(logging.INFO)
def _change_event(**kwargs) -> LogEvent:
return LogEvent(kind="change", logger=transaction_logger, action="update", **kwargs)
def test_emit_event_falsy_return_stops_chain(capsys):
_setup_logging()
calls = []
def first(ev):
calls.append("first")
return None
def second(ev):
calls.append("second")
emit_event(_change_event(), [first, second])
assert calls == ["first"]
assert capsys.readouterr().err == ""
def test_emit_event_truthy_return_falls_back_to_default(capsys):
_setup_logging()
emit_event(_change_event(), [lambda ev: True])
assert "update" in capsys.readouterr().err
def test_emit_event_mutation_reaches_later_handlers_and_default(capsys):
_setup_logging()
calls = []
def first(ev):
calls.append("first")
ev.extra = "tgt"
return True
def second(ev):
calls.append(("second", ev.extra))
return True
emit_event(_change_event(), [first, second])
assert calls == ["first", ("second", "tgt")]
assert "tgt" in capsys.readouterr().err
def test_emit_event_handler_error_falls_back_to_default(capsys):
_setup_logging()
def boom(ev):
raise RuntimeError("broken")
emit_event(_change_event(), [boom])
assert "update" in capsys.readouterr().err
def test_diff_lines_built_lazily(monkeypatch):
def _boom(*args, **kwargs):
raise AssertionError("format_diff should not be called")
monkeypatch.setattr("kanta.logging.format_diff", _boom)
ev = _change_event(diff={"counter": 1})
emit_event(ev, [lambda ev: None]) # handled without touching the diff
monkeypatch.undo()
assert len(ev.diff_lines) == 1
assert "counter" in ev.diff_lines[0]
def test_default_emit_created_and_migrated(capsys):
_setup_logging()
emit_event(LogEvent(kind="created", logger=bootstrap_logger, filename="x.kantadb"))
emit_event(
LogEvent(
kind="migrated",
logger=migration_logger,
filename="x.kantadb",
from_version=0,
to_version=1,
migrations=["migrate_v1 (rename)"],
)
)
err = capsys.readouterr().err
assert "🛢️ x.kantadb created" in err
assert "🛢️ x.kantadb migrated v0 -> v1: migrate_v1 (rename)" in err
def test_default_emit_strips_ansi_without_color_support(capsys, monkeypatch):
"""NO_COLOR output contains no ANSI codes; FORCE_COLOR keeps them."""
_setup_logging()
monkeypatch.setenv("NO_COLOR", "1")
monkeypatch.delenv("FORCE_COLOR", raising=False)
emit_event(_change_event(diff={"counter": 1}))
err = capsys.readouterr().err
assert "\x1b[" not in err
assert "counter" in err
_setup_logging()
monkeypatch.setenv("FORCE_COLOR", "1")
monkeypatch.delenv("NO_COLOR", raising=False)
emit_event(_change_event(diff={"counter": 1}))
assert "\x1b[" in capsys.readouterr().err
@pytest.mark.asyncio
async def test_logemit_receives_transaction_events(tmp_path, format_config):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
events = []
kanta.logemit(lambda ev: events.append(ev) or True)
await kanta.open()
with kanta.transaction(action="inc", user="u1", extra="x") as data:
data.counter = 1
await kanta.close()
change = events[-1]
assert change.kind == "change"
assert change.action == "inc"
assert change.user == "u1"
assert change.extra == "x"
assert change.diff == {"counter": 1}
assert change.logger.name == "kanta.transaction"
def test_logemit_rejects_classes_and_async(tmp_path, format_config):
kanta = make_kanta(tmp_path / "test.db", Data, format_config)
class NotAFunction:
pass
with pytest.raises(TypeError):
kanta.logemit(NotAFunction)
async def ahandler(ev):
return None
with pytest.raises(TypeError):
kanta.logemit(ahandler)
def _raise(*args, **kwargs):
raise RuntimeError("formatting broken")
def test_log_change_never_raises(monkeypatch):
monkeypatch.setattr("kanta.logging.format_action_header", _raise)
log_change("update", {"counter": 1}, previous={}) # must not raise
@pytest.mark.asyncio
async def test_logging_failure_does_not_break_transaction(
tmp_path, format_config, monkeypatch
):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
kanta.logemit(_raise)
monkeypatch.setattr("kanta.logging.format_action_header", _raise)
await kanta.open()
with kanta.transaction(action="inc") as data:
data.counter = 1
await kanta.close()
kanta2 = make_kanta(path, Data, format_config)
kanta2.logemit(_raise)
monkeypatch.setattr("kanta.logging.format_action_header", _raise)
await kanta2.open()
assert kanta2.data.counter == 1
await kanta2.close()
@pytest.mark.asyncio
async def test_logfmt_failure_falls_back_to_default(tmp_path, format_config, caplog):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
@kanta.logfmt
def bad(value: str, path: str) -> str | None:
raise RuntimeError("broken")
await kanta.open()
with caplog.at_level(logging.INFO, logger="kanta.transaction"):
with kanta.transaction(action="inc", user="alice") as data:
data.counter = 1
await kanta.close()
assert kanta.data.counter == 1
assert "alice" in caplog.text # raw rendering used despite the failure
assert "counter" in caplog.text
@pytest.mark.asyncio
async def test_logmigr_failure_does_not_break_open(tmp_path, format_config):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("init", {"counter": 0}), format_config)
mod = type(sys)("test_migrations_broken_logmigr")
def migrate_v1(d, kanta):
"""Bump counter."""
d["counter"] = 2
mod.__dict__["migrate_v1"] = migrate_v1
kanta = make_kanta(path, Data, format_config, migrations=mod)
@kanta.logmigr
def bad(summary: MigrationResult) -> None:
raise RuntimeError("broken")
await kanta.open()
assert kanta.data.counter == 2
await kanta.close()
@pytest.mark.asyncio
async def test_aborted_transaction_emits_event(
tmp_path, format_config, caplog, monkeypatch
):
monkeypatch.setenv("FORCE_COLOR", "1")
monkeypatch.delenv("NO_COLOR", raising=False)
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
events = []
kanta.logemit(lambda ev: events.append(ev) or True)
await kanta.open()
with caplog.at_level(logging.WARNING, logger="kanta.transaction"):
with pytest.raises(ValueError):
with kanta.transaction(action="reset") as data:
data.counter = 99
raise ValueError("simulated failure")
await kanta.close()
aborted = events[-1]
assert aborted.kind == "aborted"
assert aborted.action == "reset"
assert aborted.level == logging.WARNING
assert isinstance(aborted.error, ValueError)
messages = [r.getMessage() for r in caplog.records if r.levelno == logging.WARNING]
assert any("\x1b[1;34mreset" in m for m in messages) # action color, no quotes
assert any(" transaction aborted: simulated failure" in m for m in messages)
assert kanta.data.counter == 0 # rolled back
@pytest.mark.asyncio
async def test_aborted_transaction_includes_resolved_user(
tmp_path, format_config, caplog
):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
@kanta.logfmt
def resolve(value: str, path: str) -> str | None:
return "Alice" if value == "u1" else None
await kanta.open()
with caplog.at_level(logging.WARNING, logger="kanta.transaction"):
with pytest.raises(ValueError):
with kanta.transaction(action="reset", user="u1", extra="exp") as data:
data.counter = 99
raise ValueError("boom")
await kanta.close()
messages = [r.getMessage() for r in caplog.records if r.levelno == logging.WARNING]
assert any("exp" in m for m in messages)
assert any(" by " in m and "Alice" in m for m in messages)
assert any(" transaction aborted: boom" in m for m in messages)
def test_event_header_covers_all_kinds():
created = LogEvent(kind="created", logger=transaction_logger, filename="x.db")
assert created.header == "🛢️ x.db created"
migrated = LogEvent(
kind="migrated",
logger=transaction_logger,
filename="x.db",
from_version=0,
to_version=1,
migrations=["migrate_v1 (rename)"],
)
assert migrated.header == "🛢️ x.db migrated v0 -> v1: migrate_v1 (rename)"
aborted = LogEvent(
kind="aborted",
logger=transaction_logger,
action="reset",
user="alice",
error=ValueError("boom"),
)
assert "transaction aborted: boom" in aborted.header
assert "alice" in aborted.header
@pytest.mark.asyncio
async def test_event_carries_kanta_instance(tmp_path, format_config):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
events = []
kanta.logemit(lambda ev: events.append(ev) or True)
await kanta.open()
with kanta.transaction(action="inc") as data:
data.counter = 1
await kanta.close()
assert events
assert all(ev.kanta is kanta for ev in events)
def test_header_is_settable_and_used_by_default_emit(capsys):
_setup_logging()
def restyle(ev):
ev.header = f"CUSTOM {ev.action}"
return True
emit_event(_change_event(diff={"counter": 1}, previous={}), [restyle])
err = capsys.readouterr().err
assert "CUSTOM update" in err
assert "counter" in err # default diff routing still applies
@pytest.mark.asyncio
async def test_ctx_reachable_from_event(tmp_path, format_config):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
kanta.ctx.connection_id = 7
seen = []
kanta.logemit(lambda ev: seen.append(ev.kanta.ctx.connection_id) or True)
await kanta.open()
with kanta.transaction(action="inc") as data:
data.counter = 1
await kanta.close()
assert seen and all(connection_id == 7 for connection_id in seen)
+109 -6
View File
@@ -1,17 +1,120 @@
import logging import logging
from kanta.logging import configure_logging, log_change import pytest
from kanta.logging import logger
from kanta.logging import (
configure_logging,
format_action_header,
log_change,
)
from kanta.tty import ESC
def test_configure_logging(): def test_format_action_header():
header = format_action_header("update", "alice", "tgt")
assert header == (
f"{ESC}1;34mupdate{ESC}0m {ESC}38;5;250mtgt{ESC}0m by {ESC}34malice{ESC}0m"
)
def test_format_action_header_action_only():
assert format_action_header("update") == f"{ESC}1;34mupdate{ESC}0m"
@pytest.fixture(autouse=True)
def _reset_kanta_loggers():
yield
for name in (
"kanta",
"kanta.transaction",
"kanta.transaction.diff",
"kanta.bootstrap",
"kanta.migration",
):
logger = logging.getLogger(name)
logger.setLevel(logging.NOTSET)
logger.propagate = True
logger.disabled = False
logger.handlers.clear()
def test_configure_logging_defaults():
configure_logging() configure_logging()
assert logger.level == logging.INFO for name in ("kanta.bootstrap", "kanta.migration", "kanta.transaction"):
logger = logging.getLogger(name)
assert logger.level == logging.NOTSET # inherits the root level
assert not logger.propagate
assert logger.handlers
def test_configure_logging_disables_specific_loggers():
configure_logging(bootstrap=False, migration=False, transaction=False)
assert logging.getLogger("kanta.bootstrap").disabled
assert logging.getLogger("kanta.migration").disabled
assert logging.getLogger("kanta.transaction").disabled
def test_configure_logging_skiproot_false_routes_via_root():
configure_logging(skiproot=False)
for name in ("kanta.bootstrap", "kanta.migration", "kanta.transaction"):
logger = logging.getLogger(name)
assert logger.propagate
assert not logger.handlers
def _setup_logging(**kwargs):
"""Default kanta logging with the event loggers lifted to INFO.
Event loggers inherit the root level (WARNING under pytest); output
assertions need INFO.
"""
configure_logging(**kwargs)
for name in ("kanta.bootstrap", "kanta.migration", "kanta.transaction"):
logging.getLogger(name).setLevel(logging.INFO)
def test_log_change_no_diff(capsys): def test_log_change_no_diff(capsys):
logger.handlers.clear() _setup_logging()
configure_logging()
log_change("test", {}) log_change("test", {})
captured = capsys.readouterr() captured = capsys.readouterr()
assert "test" in captured.err assert "test" in captured.err
def test_log_change_appends_extra_string(capsys, monkeypatch):
monkeypatch.setenv("FORCE_COLOR", "1")
monkeypatch.delenv("NO_COLOR", raising=False)
_setup_logging()
log_change("export", {}, extra="mydb.db")
captured = capsys.readouterr()
assert "export" in captured.err
assert f"{ESC}38;5;250mmydb.db{ESC}0m" in captured.err
def test_log_change_log_diff_false(capsys, monkeypatch):
_setup_logging()
def _boom(*args, **kwargs):
raise AssertionError("format_diff should not be called")
monkeypatch.setattr("kanta.logging.format_diff", _boom)
log_change("update", {"counter": 5}, previous={}, log_diff=False)
captured = capsys.readouterr()
assert "update" in captured.err
assert "counter" not in captured.err
def test_configure_logging_diff_false(capsys):
_setup_logging(diff=False)
log_change("update", {"counter": 5}, previous={})
captured = capsys.readouterr()
assert "update" in captured.err
assert "counter" not in captured.err
def test_configure_logging_diff_true_reenables(capsys):
_setup_logging(diff=False)
configure_logging(diff=True)
logging.getLogger("kanta.transaction").setLevel(logging.INFO)
log_change("update", {"counter": 5}, previous={})
captured = capsys.readouterr()
assert "counter" in captured.err
+190 -16
View File
@@ -1,53 +1,227 @@
from types import ModuleType from types import ModuleType, SimpleNamespace
from kanta.migrate import MigrationRegistry import pytest
from kanta.exceptions import DatabaseError
from kanta.migrations import Migrations
class _DummyKanta:
def __init__(self):
self.ctx = SimpleNamespace()
def test_register_and_apply(): def test_register_and_apply():
reg = MigrationRegistry() reg = Migrations()
kanta = _DummyKanta()
@reg.register @reg.register
def migrate_v1(d, ctx): def migrate_v1(d, kanta):
d["version"] = 1 d["version"] = 1
@reg.register @reg.register
def migrate_v2(d, ctx): def migrate_v2(d, kanta):
d["version"] = 2 d["version"] = 2
state = {} state = {}
new_ver = reg.apply(state, current_version=0, silent=True) result = reg.apply(state, current_version=0, kanta=kanta)
assert new_ver == 2 assert result.version == 2
assert state["version"] == 2 assert state["version"] == 2
def test_no_migrations_needed(): def test_no_migrations_needed():
reg = MigrationRegistry() reg = Migrations()
kanta = _DummyKanta()
@reg.register @reg.register
def migrate_v1(d, ctx): def migrate_v1(d, kanta):
d["x"] = 1 d["x"] = 1
state = {"x": 1} state = {"x": 1}
new_ver = reg.apply(state, current_version=1, silent=True) result = reg.apply(state, current_version=1, kanta=kanta)
assert new_ver == 1 assert result.version == 1
def test_from_module(): def test_from_module():
mod = ModuleType("fake_migrations") mod = ModuleType("fake_migrations")
kanta = _DummyKanta()
def migrate_v1(d, ctx): def migrate_v1(d, kanta):
d["v"] = 1 d["v"] = 1
def migrate_v2(d, ctx): def migrate_v2(d, kanta):
d["v"] = 2 d["v"] = 2
mod.__dict__["migrate_v1"] = migrate_v1 mod.__dict__["migrate_v1"] = migrate_v1
mod.__dict__["migrate_v2"] = migrate_v2 mod.__dict__["migrate_v2"] = migrate_v2
reg = MigrationRegistry.from_module(mod) reg = Migrations.from_module(mod)
assert reg.dbver == 2 assert reg.dbver == 2
state = {} state = {}
new_ver = reg.apply(state, current_version=0, silent=True) result = reg.apply(state, current_version=0, kanta=kanta)
assert new_ver == 2 assert result.version == 2
assert state["v"] == 2 assert state["v"] == 2
def test_migrations_can_use_kanta_ctx():
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v1(d, kanta):
kanta.ctx.source = "migration"
d["source"] = kanta.ctx.source
state = {}
result = reg.apply(state, current_version=0, kanta=kanta)
assert result.version == 1
assert state["source"] == "migration"
assert kanta.ctx.source == "migration"
def test_migration_can_omit_kanta_argument():
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v1(d):
d["x"] = 1
state = {}
result = reg.apply(state, current_version=0, kanta=kanta)
assert result.version == 1
assert state["x"] == 1
def test_version_too_new():
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v1(d):
d["x"] = 1
with pytest.raises(
DatabaseError,
match="Database version v2 is newer than the highest supported version v1",
):
reg.apply({}, current_version=2, kanta=kanta)
def test_version_too_old():
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v3(d):
d["x"] = 3
with pytest.raises(
DatabaseError,
match="Database version v1 is older than the minimum supported version v2",
):
reg.apply({}, current_version=1, kanta=kanta)
def test_missing_middle_migration_is_skipped():
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v1(d):
d["x"] = 1
@reg.register
def migrate_v3(d):
d["y"] = 3
state = {"x": 1}
result = reg.apply(state, current_version=1, kanta=kanta)
assert result.version == 3
assert state["x"] == 1
assert state["y"] == 3
def test_old_migrations_deleted_current_supported():
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v3(d):
d["x"] = 3
state = {"x": 2}
result = reg.apply(state, current_version=2, kanta=kanta)
assert result.version == 3
assert state["x"] == 3
def test_apply_returns_change_information():
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v1(d):
"""Set x."""
d["x"] = 1
@reg.register
def migrate_v2(d):
"""No-op."""
pass
@reg.register
def migrate_v3(d):
"""Set y."""
d["y"] = 3
result = reg.apply({}, current_version=0, kanta=kanta)
assert result.version == 3
assert len(result.migrations) == 3
assert result.migrations[0].name == "migrate_v1"
assert result.migrations[0].description == "Set x"
assert result.migrations[0].changed is True
assert result.migrations[0].diff == {"x": 1}
assert result.migrations[1].name == "migrate_v2"
assert result.migrations[1].description == "No-op"
assert result.migrations[1].changed is False
assert result.migrations[1].diff is None
assert result.migrations[2].name == "migrate_v3"
assert result.migrations[2].description == "Set y"
assert result.migrations[2].changed is True
assert result.migrations[2].diff == {"y": 3}
def test_description_defaults_to_version_when_no_docstring():
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v1(d):
d["x"] = 1
result = reg.apply({}, current_version=0, kanta=kanta)
assert result.migrations[0].description == "v1"
def test_report_fields():
from kanta import MigrationReport
reg = Migrations()
kanta = _DummyKanta()
@reg.register
def migrate_v1(d):
d["x"] = 1
report = reg.apply({"x": 0}, current_version=0, kanta=kanta)
assert isinstance(report, MigrationReport)
assert report.original == 0
assert report.version == 1
assert [m.name for m in report.applied] == ["migrate_v1"]
# Deprecated alias still works.
assert report.migrations is report.applied
+3 -2
View File
@@ -82,8 +82,9 @@ async def test_transaction_mtime_false_preserves_mtime(tmp_path, format_config):
continue continue
records.append(serializer.decode(payload, type=ChangeRecord)) records.append(serializer.decode(payload, type=ChangeRecord))
assert records[0].m == first_m assert records[0].a == "bootstrap"
assert records[1].m is None assert records[1].m == first_m
assert records[2].m is None
assert kanta.mtime == first_m assert kanta.mtime == first_m
+153
View File
@@ -0,0 +1,153 @@
"""Tests for Kanta read-only mode."""
import pytest
from kanta.exceptions import DataIntegrityError, FileLockError
from kanta.serialization import struct_to_dict
from .support import (
Data,
EvolvableDataV2,
fixed_change,
make_kanta,
make_migrations_module,
seed_single_change,
)
@pytest.mark.asyncio
async def test_readonly_opens_existing_database(tmp_path, format_config):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("seed", {"counter": 7}), format_config)
kanta = make_kanta(path, Data, format_config)
await kanta.open(readonly=True)
assert isinstance(kanta.data, Data)
assert kanta.data.counter == 7
assert kanta._impl.readonly is True
assert kanta._impl.background_task is None
await kanta.close()
@pytest.mark.asyncio
async def test_readonly_missing_file_fails(tmp_path, format_config):
path = tmp_path / "missing.db"
kanta = make_kanta(path, Data, format_config)
with pytest.raises(FileLockError):
await kanta.open(readonly=True)
assert not path.exists()
@pytest.mark.asyncio
async def test_readonly_empty_file_fails(tmp_path, format_config):
path = tmp_path / "empty.db"
path.touch()
kanta = make_kanta(path, Data, format_config)
with pytest.raises(DataIntegrityError, match="empty"):
await kanta.open(readonly=True)
@pytest.mark.asyncio
async def test_readonly_transaction_fails(tmp_path, format_config):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("seed", {"counter": 1}), format_config)
kanta = make_kanta(path, Data, format_config)
await kanta.open(readonly=True)
with pytest.raises(DataIntegrityError, match="read-only"):
with kanta.transaction(action="inc") as data:
data.counter = 2
# In-memory state must remain unchanged.
assert kanta.data.counter == 1
await kanta.close()
@pytest.mark.asyncio
async def test_readonly_flush_fails(tmp_path, format_config):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("seed", {"counter": 1}), format_config)
kanta = make_kanta(path, Data, format_config)
await kanta.open(readonly=True)
with pytest.raises(DataIntegrityError, match="read-only"):
await kanta.flush()
await kanta.close()
@pytest.mark.asyncio
async def test_readonly_create_true_does_not_create_file(tmp_path, format_config):
path = tmp_path / "test.db"
kanta = make_kanta(path, Data, format_config)
with pytest.raises(FileLockError):
await kanta.open(create=True, readonly=True)
assert not path.exists()
@pytest.mark.asyncio
async def test_readonly_does_not_persist_changes(tmp_path, format_config):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("seed", {"counter": 1}), format_config)
original_content = path.read_bytes()
kanta = make_kanta(path, Data, format_config)
await kanta.open(readonly=True)
await kanta.close()
assert path.read_bytes() == original_content
@pytest.mark.asyncio
async def test_readonly_runs_migrations(tmp_path, format_config):
path = tmp_path / "test.db"
seed_single_change(
path,
fixed_change("seed", {"counter": 1}, version=0),
format_config,
)
def migrate_v1(data, kanta):
data.setdefault("enabled", True)
migrations = make_migrations_module("readonly_migrations", "migrate_v1", migrate_v1)
kanta = make_kanta(path, EvolvableDataV2, format_config, migrations=migrations)
await kanta.open(readonly=True)
assert kanta.data.counter == 1
# Migration ran in memory even though no change was persisted.
assert struct_to_dict(kanta.data, serializer=kanta._impl.serializer) == {
"counter": 1,
"enabled": True,
}
assert not kanta._impl.pending_changes
await kanta.close()
@pytest.mark.asyncio
async def test_readwrite_and_readonly_can_open_together(tmp_path, format_config):
path = tmp_path / "test.db"
seed_single_change(path, fixed_change("seed", {"counter": 1}), format_config)
rw = make_kanta(path, Data, format_config)
await rw.open()
ro = make_kanta(path, Data, format_config)
await ro.open(readonly=True)
assert rw.data.counter == 1
assert ro.data.counter == 1
await ro.close()
await rw.close()
+214
View File
@@ -0,0 +1,214 @@
"""Tests for retention-based database rotation (docs/rotation.md)."""
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from kanta.structs import ChangeRecord, Snapshot
from tests.support import Data, make_kanta, read_changes
pytestmark = pytest.mark.asyncio
DAY = timedelta(days=1)
T0 = datetime(2026, 1, 1, tzinfo=UTC)
def make_clock(cell: list[datetime]):
def clock() -> datetime:
return cell[0]
return clock
async def write_history(path: Path, format_config, days: list[int]) -> None:
"""Write one change per day offset (relative to T0) with a fake clock."""
cell = [T0 + (days[0] - 1) * DAY] # bootstrap predates all history
kanta = make_kanta(path, Data, format_config)
kanta.clock(make_clock(cell))
await kanta.open(log=False)
for day in days:
cell[0] = T0 + day * DAY
with kanta.transaction(f"day{day}", log=False) as data:
data.counter += 1
await kanta.flush()
await kanta.close()
def read_all(path: Path, format_config):
"""All records (changes and snapshots) in file order."""
_, serializer_cls = format_config
serializer = serializer_cls()
framer = serializer.framer_cls()
out = []
for is_snapshot, payload, _, _ in framer.iter_records(path.read_bytes(), 0):
out.append(
serializer.decode(payload, type=Snapshot if is_snapshot else ChangeRecord)
)
return out
def rotated_files(path: Path) -> list[Path]:
return sorted(path.parent.glob(f"{path.stem}@*.kantadb"))
async def test_rotation_splits_history(tmp_path, format_config):
path = tmp_path / "data.kantadb"
await write_history(path, format_config, days=[-40, -20, -5])
cell = [T0]
kanta = make_kanta(path, Data, format_config, retention=30 * DAY)
kanta.clock(make_clock(cell))
await kanta.open(log=False)
assert kanta.data.counter == 3
await kanta.close()
rotated = rotated_files(path)
assert len(rotated) == 1
# Main file: leading snapshot (ts = last dropped record), the retained
# changes, and no final snapshot (too few retained changes).
records = read_all(path, format_config)
assert isinstance(records[0], Snapshot)
assert records[0].ts == T0 - 40 * DAY
assert records[0].state["counter"] == 1
changes = [r for r in records if isinstance(r, ChangeRecord)]
assert [c.a for c in changes] == ["day-20", "day-5"]
# Rotated file holds exactly the dropped history, ending at the last
# dropped record whose ts matches the filename.
stamp = (T0 - 40 * DAY).strftime("%Y%m%dT%H%M%SZ")
assert rotated[0].name == f"data@{stamp}.kantadb"
rrecords = read_all(rotated[0], format_config)
assert [r.a for r in rrecords] == ["bootstrap", "day-40"]
async def test_rotation_reopens_cleanly_and_does_not_rerotate(tmp_path, format_config):
path = tmp_path / "data.kantadb"
await write_history(path, format_config, days=[-40, -5])
cell = [T0]
for expected_changes in (["day-5"], ["day-5"]):
kanta = make_kanta(path, Data, format_config, retention=30 * DAY)
kanta.clock(make_clock(cell))
async with kanta:
assert kanta.data.counter == 2
assert [c.a for c in read_changes(path, format_config)] == expected_changes
# Second open found a file whose history already fits the window.
assert len(rotated_files(path)) == 1
async def test_rotation_noop_when_retention_covers_all(tmp_path, format_config):
path = tmp_path / "data.kantadb"
await write_history(path, format_config, days=[-5])
before = path.read_bytes()
cell = [T0]
kanta = make_kanta(path, Data, format_config, retention=30 * DAY)
kanta.clock(make_clock(cell))
async with kanta:
assert kanta.data.counter == 1
assert rotated_files(path) == []
assert path.read_bytes() == before
async def test_rotation_aged_out_database_reduces_to_single_snapshot(
tmp_path, format_config
):
path = tmp_path / "data.kantadb"
await write_history(path, format_config, days=[-40, -35])
cell = [T0]
kanta = make_kanta(path, Data, format_config, retention=30 * DAY)
kanta.clock(make_clock(cell))
async with kanta:
assert kanta.data.counter == 2
records = read_all(path, format_config)
assert len(records) == 1
assert isinstance(records[0], Snapshot)
assert records[0].state["counter"] == 2
# Opening again must not rotate the snapshot-only file.
before = path.read_bytes()
kanta = make_kanta(path, Data, format_config, retention=30 * DAY)
kanta.clock(make_clock(cell))
async with kanta:
assert kanta.data.counter == 2
assert path.read_bytes() == before
assert len(rotated_files(path)) == 1
async def test_rotation_validates_against_internal_snapshots(tmp_path, format_config):
path = tmp_path / "data.kantadb"
cell = [T0 - 40 * DAY]
kanta = make_kanta(path, Data, format_config)
kanta.clock(make_clock(cell))
await kanta.open(log=False)
with kanta.transaction("old", log=False) as data:
data.counter = 1
await kanta.flush()
kanta.request_snapshot()
kanta._impl.maybe_snapshot()
cell[0] = T0 - 1 * DAY
with kanta.transaction("new", log=False) as data:
data.counter = 2
await kanta.flush()
await kanta.close()
cell[0] = T0
kanta = make_kanta(path, Data, format_config, retention=30 * DAY)
kanta.clock(make_clock(cell))
async with kanta:
assert kanta.data.counter == 2
records = read_all(path, format_config)
assert isinstance(records[0], Snapshot)
assert records[0].state["counter"] == 1
assert [r.a for r in records if isinstance(r, ChangeRecord)] == ["new"]
@pytest.mark.parametrize("name", ["data", "data.db", "data.kantadb"])
async def test_rotated_naming_normalizes_extension(tmp_path, format_config, name):
path = tmp_path / name
await write_history(path, format_config, days=[-40, -5])
cell = [T0]
kanta = make_kanta(path, Data, format_config, retention=30 * DAY)
kanta.clock(make_clock(cell))
async with kanta:
pass
stamp = (T0 - 40 * DAY).strftime("%Y%m%dT%H%M%SZ")
assert (tmp_path / f"data@{stamp}.kantadb").exists()
async def test_retention_accepts_int_days(tmp_path, format_config):
path = tmp_path / "data.kantadb"
await write_history(path, format_config, days=[-40, -5])
cell = [T0]
kanta = make_kanta(path, Data, format_config, retention=30)
kanta.clock(make_clock(cell))
async with kanta:
assert kanta.data.counter == 2
assert len(rotated_files(path)) == 1
assert [c.a for c in read_changes(path, format_config)] == ["day-5"]
async def test_rotation_disabled_by_default(tmp_path, format_config):
path = tmp_path / "data.kantadb"
await write_history(path, format_config, days=[-40, -5])
before = path.read_bytes()
cell = [T0]
kanta = make_kanta(path, Data, format_config)
kanta.clock(make_clock(cell))
async with kanta:
assert kanta.data.counter == 2
assert rotated_files(path) == []
assert path.read_bytes() == before
+17
View File
@@ -32,3 +32,20 @@ def test_force_writes():
f = FakeFile() f = FakeFile()
ss.maybe_write(f, 1, {"x": 1}) ss.maybe_write(f, 1, {"x": 1})
assert len(f.written) == 1 assert len(f.written) == 1
def test_force_bypasses_min_diffs():
class FakeFile:
def __init__(self):
self.written = []
self.is_open = True
def write(self, data: bytes):
self.written.append(data)
ss = SnapshotState(min_diffs=100)
ss.record_changes(5)
ss.request_force()
f = FakeFile()
ss.maybe_write(f, 1, {"x": 1})
assert len(f.written) == 1
+74
View File
@@ -0,0 +1,74 @@
import pytest
from kanta.tty import ESC, Colors, Line, colors, displaywidth, pad, strip_ansi
def test_strip_ansi():
assert strip_ansi(f"{ESC}1;34mhello{ESC}0m") == "hello"
def test_displaywidth_plain_and_ansi():
assert displaywidth("hello") == 5
assert displaywidth(f"{ESC}38;5;226mhi{ESC}0m") == 2
def test_displaywidth_wide_and_combining_chars():
assert displaywidth("你好") == 4
assert displaywidth("🚀") == 2
assert displaywidth("") == 1
def test_pad():
assert pad("ab", 4) == "ab "
assert pad("ab", 4, align="right") == " ab"
assert pad("ab", 5, align="center") == " ab "
assert pad("abcdef", 4) == "abcdef"
assert pad("你好", 6) == "你好 "
def test_line_plain_and_str_conversion():
assert str(Line()("n=", 42)) == "n=42"
def test_line_color_auto_resets_on_next_call():
assert str(Line().user("Alice")(" by ")) == f"{ESC}34mAlice{ESC}0m by "
def test_line_str_restores_active_color():
assert str(Line().user("Alice")) == f"{ESC}34mAlice{ESC}0m"
def test_line_same_color_not_reemitted():
assert str(Line().user("a").user("b")) == f"{ESC}34mab{ESC}0m"
def test_line_transition_folds_reset_into_one_sequence():
# bold blue -> plain blue: the bold clear rides in the same sequence
assert str(Line().action("a").user("b")) == f"{ESC}1;34ma{ESC}0;34mb{ESC}0m"
def test_line_unknown_color_raises():
with pytest.raises(AttributeError, match="unknown color"):
Line().nosuchcolor("x")
def test_line_palette_addition(monkeypatch):
monkeypatch.setattr(colors, "session", "38;5;226", raising=False)
assert str(Line().session("3")) == f"{ESC}38;5;226m3{ESC}0m"
def test_line_palette_override_takes_effect(monkeypatch):
monkeypatch.setattr(colors, "user", "36")
assert str(Line().user("x")) == f"{ESC}36mx{ESC}0m"
def test_line_custom_palette():
palette = Colors()
palette.brand = "35"
assert str(Line(palette).brand("x")) == f"{ESC}35mx{ESC}0m"
def test_line_width_and_align():
assert str(Line()("ab", width=4)) == "ab "
assert str(Line()("ab", width=4, align="right")) == " ab"
assert str(Line().user("ab", width=4)) == f"{ESC}34mab {ESC}0m"
+116
View File
@@ -0,0 +1,116 @@
"""Tests for the @kanta.validate integrity-validation callbacks."""
import pytest
from tests.support import Data, make_kanta
pytestmark = pytest.mark.asyncio
async def test_validate_passes_on_valid_data(tmp_path, format_config):
kanta = make_kanta(tmp_path / "d.kantadb", Data, format_config)
calls = []
@kanta.validate
def check(data: Data):
calls.append(data.counter)
assert data.counter >= 0
async with kanta:
with kanta.transaction("inc", log=False) as data:
data.counter = 1
assert calls # ran during bootstrap/open and the transaction
async def test_validate_failure_rolls_back_transaction(tmp_path, format_config):
kanta = make_kanta(tmp_path / "d.kantadb", Data, format_config)
@kanta.validate
def check(data: Data):
if data.counter < 0:
raise ValueError("counter must not go negative")
await kanta.open(log=False)
with pytest.raises(ValueError, match="negative"):
with kanta.transaction("dec", log=False) as data:
data.counter = -1
assert kanta.data.counter == 0 # rolled back
await kanta.close()
# The invalid change never reached the history.
kanta2 = make_kanta(tmp_path / "d.kantadb", Data, format_config)
async with kanta2:
assert kanta2.data.counter == 0
async def test_validate_runs_on_open_after_replay(tmp_path, format_config):
kanta = make_kanta(tmp_path / "d.kantadb", Data, format_config)
async with kanta:
with kanta.transaction("set", log=False) as data:
data.counter = 5
kanta2 = make_kanta(tmp_path / "d.kantadb", Data, format_config)
seen = []
@kanta2.validate
def check(data: Data):
seen.append(data.counter)
async with kanta2:
pass
assert 5 in seen
async def test_validate_failure_aborts_open(tmp_path, format_config):
kanta = make_kanta(tmp_path / "d.kantadb", Data, format_config)
async with kanta:
with kanta.transaction("set", log=False) as data:
data.counter = 5
kanta2 = make_kanta(tmp_path / "d.kantadb", Data, format_config)
@kanta2.validate
def check(data: Data):
raise ValueError("always inconsistent")
with pytest.raises(ValueError, match="inconsistent"):
await kanta2.open(log=False)
# The failed open released the file: a fresh instance can open it.
kanta3 = make_kanta(tmp_path / "d.kantadb", Data, format_config)
async with kanta3:
assert kanta3.data.counter == 5
async def test_multiple_validators_stop_at_first_failure(tmp_path, format_config):
kanta = make_kanta(tmp_path / "d.kantadb", Data, format_config)
calls = []
@kanta.validate
def first(data: Data):
calls.append("first")
if data.counter > 1:
raise ValueError("too big")
@kanta.validate
def second(data: Data):
calls.append("second")
await kanta.open(log=False)
calls.clear()
with pytest.raises(ValueError, match="too big"):
with kanta.transaction("bump", log=False) as data:
data.counter = 2
assert calls == ["first"]
await kanta.close()
async def test_validate_rejects_async_callback(tmp_path, format_config):
kanta = make_kanta(tmp_path / "d.kantadb", Data, format_config)
with pytest.raises(TypeError, match="must not be async"):
@kanta.validate
async def check(data: Data):
pass