# Session handoff — 2026-05-08 (5) Fifth handover for what's been a long day. The previous session (handoff-4) shipped pretty-table rendering, the B2/C2 column ops, and **designed** ADR-0017. This session implemented ADR-0017 in full, drafted and implemented a new ADR-0018 covering auto-fill semantics for `serial` and `shortid`, and landed a small parser-error tiny-win along the way. The user is busy for a while; the next agent session can pick up several well-scoped tasks listed in §"Independent work" without further input. ## State at handoff **Branch:** `main`. Working tree clean. **4 commits ahead of `origin/main`** (the 1 ADR design commit from handoff-4's last action plus 3 from this session). Push remains the user's call. Commits since handoff-4: ``` 5bb0a14 ADR-0018 implementation: auto-fill contracts for serial and shortid 7dfa718 parser: structural error rendering, source echo, and caret pointer 00947b9 ADR-0017 implementation: per-cell type-change with override flags 545cbf5 Handoff doc for end of 2026-05-08 (#4) c3e5f90 ADR-0017 + ADR-0002 amendment: type-change compatibility + engine-agnostic posture ``` **Tests:** **534 passing, 0 failing, 0 skipped** (up from 449 at handoff-4's baseline; +85 over this session). Test counts per phase: - ADR-0017 implementation: +68 (449 → 517) - Parser tiny-win: +2 (517 → 519) - ADR-0018 implementation: +15 (519 → 534) **Clippy:** clean with `nursery` lints enabled. **Release build:** ~7.2 MB single binary (up ~100 KB from handoff-4's 7.1 MB; the increase is the type_change matrix module and ADR-0018 auto-fill paths). ## What's implemented (delta vs. handoff-4) The previous handoff covered: Iter 1–6 of track 2, pretty-table rendering, B2/C2 column ops, optional `to`/`table` parser polish, and silent-rebuild banner suppression. This session adds: ### ADR-0017 (column type-change compatibility) — implemented Replaces the placeholder "trust STRICT" body of `do_change_column_type` with the full per-cell transformer matrix from ADR-0017. New module `src/type_change.rs` (~620 lines + 56 unit tests) carrying: - `CellOutcome { Clean(Value) | Lossy { new, reason } | Incompatible { reason } }` plus `transform_cell` covering every entry in ADR-0017 §3. - `static_refusal` for same-type / blob / date↔datetime / cross-domain refusals. `change column [in] [table] : ()` now accepts `--force-conversion` (accept lossy) and `--dont-convert` (skip the entire client-side layer; let the engine's STRICT typing decide). Mutually exclusive at parse time. Refusal preconditions per ADR-0017 §4: - Outbound FK (column is a child-side FK): refused outright. - Inbound FK (column is parent-side / referenced): refused only when `old_ty.fk_target_type() != new_ty.fk_target_type()`. - Post-transformation uniqueness check for any column that carries a UNIQUE constraint in the new schema (PK + ADR- 0018's added serial/shortid). Diagnostic refusals render through ADR-0016's pretty-table renderer — bordered, capped at 100 rows with `… and N more` inside the box, identifying rows by their PK value(s) per the ADR-0017 §7 amendment we added (PK identifiers replace positional row indices, since SQLite returns rows unordered). `[client-side]` success note (§6) fires when any cell was non-identity transformed; lossy variant adds the lossy count under `--force-conversion`. ADR-0017 §3 was amended in place to add `serial → int` as an always-clean matrix entry (it was missing despite §4.1 treating it as the canonical PK conversion). ### Parser: structural error rendering + source echo + caret The old `humanise()` rendered chumsky's terse default ("found 'i' expected ':' (near `i`)") as-is and added a not-helpful `(near `X`)` suffix. Now `humanise()` reads the structured `RichReason::ExpectedFound`, lists the `expected` patterns in plain prose, prefixes the consumed context, and produces messages like: ``` parse error: after `change column Rich`, expected `:`, found `in` ``` `dispatch_dsl` additionally echoes the source line on parse failure (matching the success path's "running: …") and prints a `^` caret under the failure position. **Known limit captured for future work**: chumsky combinators in `keyword_ci` emit `Rich::custom` errors on mismatch, which are opaque to chumsky's choice-aggregation machinery. Result: errors like "expected `data` or `table`" (bison-equivalent) aren't yet possible — only one alternative shows up. A structural fix to keyword_ci would aggregate properly. Deferred to a future "parser-as- source-of-truth" ADR (covered in §"Pending" below). ### ADR-0018 (auto-fill contracts for serial and shortid) — designed and implemented User noticed three asymmetric gaps during ADR-0017 testing: 1. `serial` was restricted to single-column PK. Other RDBMS (PostgreSQL `SEQUENCE`, MySQL `AUTO_INCREMENT`) don't have this restriction; ours was an artefact of SQLite's only free auto-increment mechanism (`INTEGER PRIMARY KEY` rowid alias) leaking into the user-facing surface. 2. `text → shortid` round-trip worked end-to-end (per ADR-0017's matrix); `int → serial` was statically refused. 3. `add column T: x (shortid)` on a non-empty table left existing rows NULL — violating the design contract that shortids are unique non-null identifiers. ADR-0018 generalises both auto-generated types with the unifying principle: *auto-generated column types honour their generation contract on every path that creates or transitions the column.* Concretely: - **`serial` is no longer PK-restricted.** Non-PK serial columns get an emitted UNIQUE constraint and use application-side `MAX(col) + 1` at INSERT time. PK case unchanged (rowid alias). Implementation switch hidden per ADR-0002. - **`shortid` auto-fill at column-materialisation time.** `add column T: x (shortid)` on a non-empty table now generates fresh shortids for existing rows in the same rebuild transaction. `change column → shortid` does the same for null cells. - **`int → serial` joins the matrix** as always-clean identity. Other source types refused with a route-via- int hint. - **`change column → serial` auto-fills null cells** with sequence values continuing from `MAX + 1`. - **UNIQUE story**: non-PK serial / shortid gain UNIQUE on creation/conversion. Reverse direction (`serial → int`, `shortid → text`) leaves UNIQUE in place — user can drop it later when the constraint-management surface lands (C3-track work, deferred). ADR-0018 implementation pulled C3 partially forward: `schema_to_ddl` gains UNIQUE-clause emission, `read_schema` gains UNIQUE detection via `pragma_index_list` / `pragma_index_info`, and `ColumnSchema` (persistence) gains a `unique: bool` field that survives the YAML round- trip. The user-facing constraint surface (`add unique` syntax, drop/rename UNIQUE, multi-column UNIQUE) stays deferred — only the internal infrastructure required by the auto-generated type contracts landed. `[client-side]` notes extended: when both ADR-0017 transformation AND ADR-0018 auto-fill apply in the same operation, two distinct note lines emit (e.g., `change column T: x (shortid)` from text where some cells had to be validated and others auto-filled). `AddColumnResult` is a new return type carrying pre- rendered `[client-side]` note lines for the new auto-fill paths. ### Engine-vocabulary cleanup While in `do_add_column`, fixed an existing user-visible string that named "SQLite's ALTER TABLE" — an ADR-0002 posture violation that pre-dated this session. The refusal it lived in was being lifted anyway as part of ADR-0018, so the leak went with it. A broader engine-name sweep is listed in §"Independent work" below. ## ADR index (read these before touching the related areas) ``` 0000 Record architecture decisions (process) 0001 Language and TUI framework (Rust + Ratatui) 0002 Database engine — User-facing posture (no engine name in user-visible strings; amended in handoff-4's session) 0003 Input modes and command dispatch 0004 Project file format — amended by 0015 0005 Column type vocabulary — definition of `serial` generalised by ADR-0018 (no longer restricted to PK; implementation hidden) 0006 Undo snapshots and replay log (deferred) 0007 Sharing and export — amended by 0015 amendment 1 0008 Testing approach 0009 DSL command syntax conventions 0010 Database access via worker thread — load-bearing for ADR-0018 §5's MAX+1 INSERT path safety 0011 FK column type compatibility 0012 Internal metadata for user-facing column types 0013 Relationships, naming, and rebuild-table strategy — primitive carries every auto-fill-on-rebuild case 0014 Data operations, value literals, and auto-show — INSERT-time auto-fill amended by ADR-0018 §5 0015 Project storage runtime — ColumnSchema gained `unique: bool` for ADR-0018's round-trip (no migration needed; older project files default to `unique: false`) 0016 Pretty table rendering for data and structure views — used by ADR-0017's diagnostic tables 0017 Column type-change compatibility — IMPLEMENTED (this session). §3 + §7 amended in place for serial→int matrix entry and PK-based row identifiers. §3 + §4.3 further amended by ADR-0018 for int→serial entry and uniqueness-check extension. 0018 Auto-fill contracts for serial and shortid columns — IMPLEMENTED (this session). Generalises serial beyond PK; tightens shortid contract; pulls forward internal UNIQUE infrastructure. ``` ## Pending — proposed next moves (in order) ### 1. Independent work for next session — see dedicated section below This is the substantive output for an unattended agent session. Three Tier-A and two Tier-B items are detailed in §"Independent work for next session". ### 2. Friendly error layer (H1) — needs a small ADR first ADR-0002's user-facing posture commits to never exposing engine error text verbatim. The current friendly-message helper just calls `Display`. ADR-0017's `--dont-convert` path has a tiny local wrapper (`friendly_change_column_engine_error`) that recognises common kinds — when H1 lands, that helper folds into the broader translator. ADR scope: defining the translation mapping (which engine error patterns map to which user- facing wording), how to surface FK / NOT NULL / type- mismatch errors symmetrically. Probably 200 lines of code + tests once the ADR settles. ### 3. Parser-as-source-of-truth ADR Discussed in this session: chumsky gives us structural information (expected sets, span-tagged AST, partial parses on failure) we're not extracting. That feeds H1a (syntax help in parse errors), I3 (tab completion), I4 (syntax highlighting), and on-the-fly error squiggles. The parser tiny-win this session was a down payment; the broader ADR maps out what we extract from one source (chumsky's parse output) to drive each affordance. The specific keyword_ci structural-error rework (so "expected `data` or `table`"-style messages aggregate across choice alternatives) is the load-bearing piece. ### 4. Query DSL ADR + implementation Biggest remaining design piece. Earlier discussion landed on extending `show data` into a SELECT-style command with WHERE / projection / order; expose generated SQL as a pedagogical hook; bundle C5a's complex WHERE into one coherent feature. Then QA1 (EXPLAIN QUERY PLAN) becomes meaningful. ### 5. Bigger UX projects - V4 (session log + Markdown export). - V1/V2 pretty-rendering refinements (relationship rendering ADR — the "two structures + arrow" view). - V3 (ER diagram export). ## Independent work for next session These are well-scoped tasks an agent can pick up and finish without user input. Each is sized to fit in one session. ### A1. CI workflow (TT5) **Scope:** single GitHub Actions YAML at `.github/workflows/ci.yml`. Cross-platform Linux / macOS / Windows; `cargo test` + `cargo clippy --all-targets -- -D warnings`. Locks in the 534-test green baseline. **Why independent:** no design questions, no codebase integration. Standard Rust CI template adapted to this project's nursery-clippy posture. **Done when:** workflow file exists, syntax-validated, runs on the next push to `main`. Local verification not strictly required but `act` (if installed) can simulate. **Watch out for:** the `bundled` feature on `rusqlite` means SQLite is statically linked; no system-package install step needed. `tokio` works on all three platforms unchanged. **Estimated:** 1–2 hours. ### A2. Engine-name audit (ADR-0002 posture sweep) **Scope:** grep error messages and other user-facing strings across `src/` for "SQLite", "STRICT", "PRAGMA", "rusqlite", "ALTER TABLE", "CAST" (selectively — `CAST` is a legitimate SQL keyword users will encounter, only a problem when prescriptive). Replace with abstract "the database" / "the engine" phrasing per ADR-0002. **Why independent:** mechanical, well-defined. ADR-0002's "User-facing posture" section is the spec. **Where to look:** - `DbError` variants — `Sqlite { message }` carries engine-vocabulary; check whether `friendly_message()` needs upgrading. - Help text in `app.rs:1100-1200` area. - Error messages constructed via `format!` with `Err(...)` / `DbError::Unsupported(...)` — search for these. - Unsupported-feature refusals. **Done when:** zero matches for "SQLite" / "STRICT" / "PRAGMA" / "rusqlite" in user-reachable strings, AND the test suite still green. Code comments and ADR prose are fair game (they explicitly may name the engine — see ADR-0002). **Watch out for:** `rusqlite::Error::*` variant names that appear in formatted error messages — those leak the crate name. Replace with a switch on the error kind. **Estimated:** 1–2 hours. ### A3. `replay` command (U4) **Scope:** new DSL command `replay ` that reads a file (typically `history.log` or a `.commands` file) and dispatches each non-comment, non-blank line through the existing DSL pipeline. On a per-line failure, abort the replay and report `replay failed at line N: `. On success, report `replay complete: N command(s)`. **Why independent:** small, well-bounded. The DSL pipeline already exists; this just feeds it lines from a file. **Implementation sketch:** 1. Parser: `replay` keyword followed by a quoted or bare path. The path lexing might need a small new helper (current parser doesn't have a "file path" terminal). 2. Command AST: `Command::Replay { path: String }`. 3. Runtime: read file, iterate lines, parse-and-execute each, abort on first failure. Probably best kept transactional at the file level (no individual command commits if any later one fails) — but that's a design question worth flagging in the implementation. **Default to "stop on first error, report line number, don't roll back"**: matches the "I'm replaying my history" mental model where partial replay is a recoverable state. 4. AppEvent + handler for replay outcome. 5. Tests: happy path (3-line replay), failure-mid-replay (reports line number + stops), empty file, blank lines skipped, comment lines (`# ...`) skipped. **Watch out for:** ADR-0015's history.log format — entries are append-only DSL command lines. `replay history.log` on a project should reproduce its current state if started from an empty database. That's the implicit invariant the test suite should prove. **Estimated:** 3–4 hours. ### B1. Update help text for ADR-0017 + ADR-0018 features **Scope:** the in-app `help` command's output (in `app.rs`, the `do_help` or similar function around line 1100–1200) shows DSL command shapes. ADR-0017 added `--force- conversion` and `--dont-convert` flags (already added to help). ADR-0018 changed semantics of `add column ... (serial|shortid)` on non-empty tables (now auto-fills existing rows + emits UNIQUE) — this isn't called out anywhere user-facing. **Why independent:** the ADRs spell out the behaviour; the help text just needs to surface it. **Suggested additions:** - `add column ... (serial|shortid)` line gains a sub-line: ` (existing rows auto-filled with sequence/generated values)`. - `change column ... (serial|shortid)` similarly. - New section "Auto-generated types" explaining serial and shortid in 3-4 lines. **Done when:** the help output describes the behaviour matching ADR-0018 + ADR-0017. Existing help-output tests pass (some may need string-matching updates). **Estimated:** 30 min. ### B2. Test gap: change_column → bool from int 0/1 **Scope:** the type_change matrix has `(Int, Bool)` per- cell-classified (clean for 0/1, incompatible otherwise). This is well-tested at the matrix unit-test level. But there's no end-to-end test in `db.rs` exercising `change column T: x (bool)` from an int column. Trivial coverage gap to fill. **Why independent:** identical pattern to existing change- column tests; just a different type pair. **Suggested test:** - `change_column_type_int_to_bool_with_zero_one_succeeds`: rows with values 0, 1, 0 → success, no client-side note expected (storage class doesn't change). - `change_column_type_int_to_bool_refuses_other_values`: row with value 2 → incompatible refusal. **Done when:** 2 new tests pass; total 536. **Estimated:** 30 min. ## Sharp edges and subtleties (delta vs. handoff-4) Carried-over edges still apply (sync `update`, worker thread, metadata transactions, rebuild-table primitive, modal infrastructure, project-switch lock dance, `[temp]` cleanup guards, persistence ordering, `DataResult` carries `column_types`, `output_render` is the only place tabular output should originate). New ones this session: - **`Type::Serial` no longer implies PK at the type system level.** ADR-0018 generalised serial. Existing references to "serial" in code comments may say "PK type" — those are stale. The non-PK serial path is active and tested. - **`add column` returns `AddColumnResult`, not `TableDescription`.** Tests that called `db.add_column(...).await.unwrap()` and used the result as a description directly need `.description` indirection. Five existing tests were updated; new tests should follow the new shape. - **`ChangeColumnTypeResult.client_side` is now `Option` where `ClientSideNote` carries `transformed`, `lossy`, `auto_filled`, `auto_fill_kind`.** When auto-fill happens (target is serial/shortid + null cells), the note fires even though `transformed` is 0. The filter `note.transformed > 0 || note.auto_filled > 0` is the canonical "should we emit a note" test. - **Non-PK serial INSERT auto-fill happens via `MAX(col)+1`.** Per ADR-0010, the worker-thread serialisation makes this safe without explicit locking. If you ever extract the worker thread or change the connection model, this is one of the things that breaks. - **`schema_to_ddl` emits inline `UNIQUE` for non-PK columns flagged unique.** PK columns aren't separately marked unique in `ReadColumn` (PK already implies it); the schema_to_ddl filter `unique && !primary_key` matters. - **`read_schema` reads UNIQUE via `pragma_index_list` filtered to `origin = 'u'`.** Compound UNIQUE constraints are deliberately ignored (ADR-0018 OOS-6 / future C3). If you ever add multi-column UNIQUE support, the detection logic needs extending. - **Parse-error messages now show grammar-derived expected/found and a consumed-context prefix.** Existing tests that asserted on the old message shape may have needed updates — none did, since the structural-error tests assert on substrings (the consumed context, the expected token). ## Repository layout (delta vs. handoff-4) ``` src/ type_change.rs — new (ADR-0017) db.rs — many additions: AddColumnResult, ChangeColumn­ TypeResult, ClientSideNote, AutoFillKind, ReadColumn.unique, read_unique_columns, schema_to_ddl UNIQUE emission, do_add_plain_column / do_add_auto_ generated_column, do_change_column_type rewrite, run_change_column_with_dry_run + fill_auto_generated_cells, generate_shortid_batch, format_auto_fill_add_note, diagnostic helpers (lossy / incompatible / collision) dsl/ parser.rs — change_column flag parsing, RichPattern-aware humanise, identifier .labelled, consumed-context rendering command.rs — ChangeColumnMode enum value.rs — validate_date / validate_datetime made pub(crate) so type_change can consume them app.rs — handle_dsl_change_column_success, handle_dsl_add_column_success, source-echo + caret on parse fail event.rs — DslChangeColumnSucceeded, DslAddColumnSucceeded output_render.rs — render_diagnostic_table public, Alignment public, numeric_alignment_for public persistence/ mod.rs — ColumnSchema.unique yaml.rs — write_column emits unique flag, RawColumn parses it csv_io.rs — test fixture updated runtime.rs — CommandOutcome::ChangeColumn + AddColumn variants docs/ adr/ 0017-column-type-change-compatibility.md — §3 (serial→int row), §7 (PK identifiers) amended 0018-auto-fill-contracts-for-serial-and-shortid.md — new (this session) README.md — indexed handoff/ 20260508-handoff-5.md — this file ``` ## How to take over 1. Read this file. 2. Read `CLAUDE.md` for the working-style rules. 3. Read `docs/requirements.md` for granular progress. 4. **If picking up an Independent work item (§A1–B2)**: read just that item plus the listed ADR section it refers to. The items are scoped to be independently tackleable. 5. **If working on H1 / Query DSL / Parser-as-source-of- truth**: start with an ADR draft. Don't implement without one — those touch enough code to warrant the discipline. 6. Run `cargo test` to confirm the 534-test green baseline. 7. `cargo clippy --all-targets` to confirm clippy-clean. 8. `cargo run --release` to see the UI. ### End-to-end smoke test (current state) Demonstrates ADR-0017 + ADR-0018 features. Replaces the handoff-4 recipe (which is now stale — `change column` under ADR-0017 emits `[client-side]` notes the previous recipe didn't show). ``` $ rm -rf /tmp/handoff5-smoke $ rdbms-playground --data-dir /tmp/handoff5-smoke # Inside the app: help -- help text (B1: extend with ADR-0018 wording) create table Customers with pk id:serial add column Customers: Name (text) add column Customers: Score (int) insert into Customers ('Alice', 10) insert into Customers ('Bob', 20) insert into Customers ('Carol', 30) show data Customers -- pretty-table render # ADR-0017 type-change with [client-side] note: change column Customers: Score (real) -- emits: -- [client-side] 3 row(s) -- were transformed before -- being stored. ... # ADR-0017 lossy refusal: change column Customers: Score (int) -- emits a bordered -- diagnostic table -- listing the lossy rows -- by PK; suggests -- --force-conversion. change column Customers: Score (int) --force-conversion -- succeeds with both -- "transformed" and -- "lossy" counts in note. # ADR-0018 add column auto-fill: add column Customers: Tag (shortid) -- emits: -- [client-side] 3 row(s) -- given auto-generated -- shortid values. ... show data Customers -- Tag column populated # ADR-0018 non-PK serial INSERT auto-fill: add column Customers: Seq (serial) -- emits another -- [client-side] note insert into Customers ('Dave', 40) -- Seq auto-fills 4 -- (MAX of existing -- 1,2,3 plus 1) # ADR-0018 int -> serial round-trip: add column Customers: Counter (int) update Customers set Counter=1 where id=1 update Customers set Counter=2 where id=2 update Customers set Counter=3 where id=3 update Customers set Counter=4 where id=4 change column Customers: Counter (serial) -- succeeds (no auto-fill -- needed since values -- are unique non-null) # ADR-0017 PK FK-cascade refinement: add column Customers: Email (text) update Customers set Email='alice@example.com' where id=1 update Customers set Email='bob@example.com' where id=2 update Customers set Email='carol@example.com' where id=3 update Customers set Email='dave@example.com' where id=4 change column Customers: id (int) -- serial -> int on PK, -- no inbound FK -> -- allowed. change column Customers: id (serial) -- int -> serial round -- trip succeeds. # Parser tiny-win demo: change column Tag in Customers: Tag (text) -- typo: column-name- -- first. Error now reads -- "after `change column -- Tag`, expected `:`, -- found `in`" with caret -- under the offending -- character. quit ``` ### Manual spot-checks worth running - `--help` lists all column ops (drop / rename / change) with their flags. - Pretty rendering kicks in for `show data` AND every schema-mutating command's auto-show. - `change column T: c (real)` succeeds and emits the `[client-side]` note for any non-empty table where the source values differ in storage class from the target. - `change column T: c (real) --force-conversion` accepts fractional → int truncation; the note carries both counts. - `change column T: c (real) --dont-convert` bypasses the client-side layer entirely (no `[client-side]` note, even if all cells transformed cleanly). - `add column T: x (shortid)` on a non-empty table fills every existing row's `x` with a generated shortid. - `add column T: x (serial)` on a non-empty table fills with 1..N. Subsequent inserts get N+1, N+2… - Non-PK serial UNIQUE: `update T set Seq=1 --all-rows` → engine refuses with a unique-violation diagnostic. - Save/load round-trip: create a non-PK serial column, quit, re-open. Read back: column is still UNIQUE. - `change column id (int)` on a `serial` PK with no inbound FKs → allowed (per ADR-0017 §4.1 refinement). - `change column id (text)` on a `serial` PK with an inbound FK → refused (per ADR-0017 §4.1 — fk_target_type would change).