rdbms-playground/docs/handoff/20260508-handoff-5.md

Session handoff — 2026-05-08 (5)

Fifth handover for what's been a long day. The previous session (handoff-4) shipped pretty-table rendering, the B2/C2 column ops, and designed ADR-0017. This session implemented ADR-0017 in full, drafted and implemented a new ADR-0018 covering auto-fill semantics for serial and shortid, and landed a small parser-error tiny-win along the way. The user is busy for a while; the next agent session can pick up several well-scoped tasks listed in §"Independent work" without further input.

State at handoff

Branch: main. Working tree clean. 4 commits ahead of origin/main (the 1 ADR design commit from handoff-4's last action plus 3 from this session). Push remains the user's call.

Commits since handoff-4:

5bb0a14 ADR-0018 implementation: auto-fill contracts for serial
        and shortid
7dfa718 parser: structural error rendering, source echo, and
        caret pointer
00947b9 ADR-0017 implementation: per-cell type-change with
        override flags
545cbf5 Handoff doc for end of 2026-05-08 (#4)
c3e5f90 ADR-0017 + ADR-0002 amendment: type-change compatibility
        + engine-agnostic posture

Tests: 534 passing, 0 failing, 0 skipped (up from 449 at handoff-4's baseline; +85 over this session). Test counts per phase:

• ADR-0017 implementation: +68 (449 → 517)
• Parser tiny-win: +2 (517 → 519)
• ADR-0018 implementation: +15 (519 → 534)

Clippy: clean with nursery lints enabled.

Release build: ~7.2 MB single binary (up ~100 KB from handoff-4's 7.1 MB; the increase is the type_change matrix module and ADR-0018 auto-fill paths).

What's implemented (delta vs. handoff-4)

The previous handoff covered: Iter 1–6 of track 2, pretty-table rendering, B2/C2 column ops, optional to/table parser polish, and silent-rebuild banner suppression. This session adds:

ADR-0017 (column type-change compatibility) — implemented

Replaces the placeholder "trust STRICT" body of do_change_column_type with the full per-cell transformer matrix from ADR-0017. New module src/type_change.rs (~620 lines + 56 unit tests) carrying:

• CellOutcome { Clean(Value) | Lossy { new, reason } | Incompatible { reason } } plus transform_cell covering every entry in ADR-0017 §3.
• static_refusal for same-type / blob / date↔datetime / cross-domain refusals.

change column [in] [table] <T>: <col> (<newtype>) now accepts --force-conversion (accept lossy) and --dont-convert (skip the entire client-side layer; let the engine's STRICT typing decide). Mutually exclusive at parse time.

Refusal preconditions per ADR-0017 §4:

• Outbound FK (column is a child-side FK): refused outright.
• Inbound FK (column is parent-side / referenced): refused only when old_ty.fk_target_type() != new_ty.fk_target_type().
• Post-transformation uniqueness check for any column that carries a UNIQUE constraint in the new schema (PK + ADR- 0018's added serial/shortid).

Diagnostic refusals render through ADR-0016's pretty-table renderer — bordered, capped at 100 rows with … and N more inside the box, identifying rows by their PK value(s) per the ADR-0017 §7 amendment we added (PK identifiers replace positional row indices, since SQLite returns rows unordered).

[client-side] success note (§6) fires when any cell was non-identity transformed; lossy variant adds the lossy count under --force-conversion.

ADR-0017 §3 was amended in place to add serial → int as an always-clean matrix entry (it was missing despite §4.1 treating it as the canonical PK conversion).

Parser: structural error rendering + source echo + caret

The old humanise() rendered chumsky's terse default ("found 'i' expected ':' (near i)") as-is and added a not-helpful (near X) suffix. Now humanise() reads the structured RichReason::ExpectedFound, lists the expected patterns in plain prose, prefixes the consumed context, and produces messages like:

parse error: after `change column Rich`, expected `:`,
found `in`

dispatch_dsl additionally echoes the source line on parse failure (matching the success path's "running: …") and prints a ^ caret under the failure position.

Known limit captured for future work: chumsky combinators in keyword_ci emit Rich::custom errors on mismatch, which are opaque to chumsky's choice-aggregation machinery. Result: errors like "expected data or table" (bison-equivalent) aren't yet possible — only one alternative shows up. A structural fix to keyword_ci would aggregate properly. Deferred to a future "parser-as- source-of-truth" ADR (covered in §"Pending" below).

ADR-0018 (auto-fill contracts for serial and shortid) — designed and implemented

User noticed three asymmetric gaps during ADR-0017 testing:

1. serial was restricted to single-column PK. Other RDBMS (PostgreSQL SEQUENCE, MySQL AUTO_INCREMENT) don't have this restriction; ours was an artefact of SQLite's only free auto-increment mechanism (INTEGER PRIMARY KEY rowid alias) leaking into the user-facing surface.
2. text → shortid round-trip worked end-to-end (per ADR-0017's matrix); int → serial was statically refused.
3. add column T: x (shortid) on a non-empty table left existing rows NULL — violating the design contract that shortids are unique non-null identifiers.

ADR-0018 generalises both auto-generated types with the unifying principle: auto-generated column types honour their generation contract on every path that creates or transitions the column. Concretely:

• serial is no longer PK-restricted. Non-PK serial columns get an emitted UNIQUE constraint and use application-side MAX(col) + 1 at INSERT time. PK case unchanged (rowid alias). Implementation switch hidden per ADR-0002.
• shortid auto-fill at column-materialisation time. add column T: x (shortid) on a non-empty table now generates fresh shortids for existing rows in the same rebuild transaction. change column → shortid does the same for null cells.
• int → serial joins the matrix as always-clean identity. Other source types refused with a route-via- int hint.
• change column → serial auto-fills null cells with sequence values continuing from MAX + 1.
• UNIQUE story: non-PK serial / shortid gain UNIQUE on creation/conversion. Reverse direction (serial → int, shortid → text) leaves UNIQUE in place — user can drop it later when the constraint-management surface lands (C3-track work, deferred).

ADR-0018 implementation pulled C3 partially forward: schema_to_ddl gains UNIQUE-clause emission, read_schema gains UNIQUE detection via pragma_index_list / pragma_index_info, and ColumnSchema (persistence) gains a unique: bool field that survives the YAML round- trip. The user-facing constraint surface (add unique syntax, drop/rename UNIQUE, multi-column UNIQUE) stays deferred — only the internal infrastructure required by the auto-generated type contracts landed.

[client-side] notes extended: when both ADR-0017 transformation AND ADR-0018 auto-fill apply in the same operation, two distinct note lines emit (e.g., change column T: x (shortid) from text where some cells had to be validated and others auto-filled).

AddColumnResult is a new return type carrying pre- rendered [client-side] note lines for the new auto-fill paths.

Engine-vocabulary cleanup

While in do_add_column, fixed an existing user-visible string that named "SQLite's ALTER TABLE" — an ADR-0002 posture violation that pre-dated this session. The refusal it lived in was being lifted anyway as part of ADR-0018, so the leak went with it. A broader engine-name sweep is listed in §"Independent work" below.

ADR index (read these before touching the related areas)

0000 Record architecture decisions (process)
0001 Language and TUI framework (Rust + Ratatui)
0002 Database engine
        — User-facing posture (no engine name in user-visible
          strings; amended in handoff-4's session)
0003 Input modes and command dispatch
0004 Project file format
        — amended by 0015
0005 Column type vocabulary
        — definition of `serial` generalised by ADR-0018 (no
          longer restricted to PK; implementation hidden)
0006 Undo snapshots and replay log (deferred)
0007 Sharing and export
        — amended by 0015 amendment 1
0008 Testing approach
0009 DSL command syntax conventions
0010 Database access via worker thread
        — load-bearing for ADR-0018 §5's MAX+1 INSERT path
          safety
0011 FK column type compatibility
0012 Internal metadata for user-facing column types
0013 Relationships, naming, and rebuild-table strategy
        — primitive carries every auto-fill-on-rebuild case
0014 Data operations, value literals, and auto-show
        — INSERT-time auto-fill amended by ADR-0018 §5
0015 Project storage runtime
        — ColumnSchema gained `unique: bool` for ADR-0018's
          round-trip (no migration needed; older project
          files default to `unique: false`)
0016 Pretty table rendering for data and structure views
        — used by ADR-0017's diagnostic tables
0017 Column type-change compatibility
        — IMPLEMENTED (this session). §3 + §7 amended in place
          for serial→int matrix entry and PK-based row
          identifiers. §3 + §4.3 further amended by ADR-0018
          for int→serial entry and uniqueness-check extension.
0018 Auto-fill contracts for serial and shortid columns
        — IMPLEMENTED (this session). Generalises serial
          beyond PK; tightens shortid contract; pulls forward
          internal UNIQUE infrastructure.

Pending — proposed next moves (in order)

1. Independent work for next session — see dedicated section below

This is the substantive output for an unattended agent session. Three Tier-A and two Tier-B items are detailed in §"Independent work for next session".

2. Friendly error layer (H1) — needs a small ADR first

ADR-0002's user-facing posture commits to never exposing engine error text verbatim. The current friendly-message helper just calls Display. ADR-0017's --dont-convert path has a tiny local wrapper (friendly_change_column_engine_error) that recognises common kinds — when H1 lands, that helper folds into the broader translator. ADR scope: defining the translation mapping (which engine error patterns map to which user- facing wording), how to surface FK / NOT NULL / type- mismatch errors symmetrically. Probably 200 lines of code

• tests once the ADR settles.

3. Parser-as-source-of-truth ADR

Discussed in this session: chumsky gives us structural information (expected sets, span-tagged AST, partial parses on failure) we're not extracting. That feeds H1a (syntax help in parse errors), I3 (tab completion), I4 (syntax highlighting), and on-the-fly error squiggles. The parser tiny-win this session was a down payment; the broader ADR maps out what we extract from one source (chumsky's parse output) to drive each affordance.

The specific keyword_ci structural-error rework (so "expected data or table"-style messages aggregate across choice alternatives) is the load-bearing piece.

4. Query DSL ADR + implementation

Biggest remaining design piece. Earlier discussion landed on extending show data into a SELECT-style command with WHERE / projection / order; expose generated SQL as a pedagogical hook; bundle C5a's complex WHERE into one coherent feature. Then QA1 (EXPLAIN QUERY PLAN) becomes meaningful.

5. Bigger UX projects

• V4 (session log + Markdown export).
• V1/V2 pretty-rendering refinements (relationship rendering ADR — the "two structures + arrow" view).
• V3 (ER diagram export).

Independent work for next session

These are well-scoped tasks an agent can pick up and finish without user input. Each is sized to fit in one session.

A1. CI workflow (TT5)

Scope: single GitHub Actions YAML at .github/workflows/ci.yml. Cross-platform Linux / macOS / Windows; cargo test + cargo clippy --all-targets -- -D warnings. Locks in the 534-test green baseline.

Why independent: no design questions, no codebase integration. Standard Rust CI template adapted to this project's nursery-clippy posture.

Done when: workflow file exists, syntax-validated, runs on the next push to main. Local verification not strictly required but act (if installed) can simulate.

Watch out for: the bundled feature on rusqlite means SQLite is statically linked; no system-package install step needed. tokio works on all three platforms unchanged.

Estimated: 1–2 hours.

A2. Engine-name audit (ADR-0002 posture sweep)

Scope: grep error messages and other user-facing strings across src/ for "SQLite", "STRICT", "PRAGMA", "rusqlite", "ALTER TABLE", "CAST" (selectively — CAST is a legitimate SQL keyword users will encounter, only a problem when prescriptive). Replace with abstract "the database" / "the engine" phrasing per ADR-0002.

Why independent: mechanical, well-defined. ADR-0002's "User-facing posture" section is the spec.

Where to look:

• DbError variants — Sqlite { message } carries engine-vocabulary; check whether friendly_message() needs upgrading.
• Help text in app.rs:1100-1200 area.
• Error messages constructed via format! with Err(...) / DbError::Unsupported(...) — search for these.
• Unsupported-feature refusals.

Done when: zero matches for "SQLite" / "STRICT" / "PRAGMA" / "rusqlite" in user-reachable strings, AND the test suite still green. Code comments and ADR prose are fair game (they explicitly may name the engine — see ADR-0002).

Watch out for: rusqlite::Error::* variant names that appear in formatted error messages — those leak the crate name. Replace with a switch on the error kind.

Estimated: 1–2 hours.

A3. replay command (U4)

Scope: new DSL command replay <path> that reads a file (typically history.log or a .commands file) and dispatches each non-comment, non-blank line through the existing DSL pipeline. On a per-line failure, abort the replay and report replay failed at line N: <error>. On success, report replay complete: N command(s).

Why independent: small, well-bounded. The DSL pipeline already exists; this just feeds it lines from a file.

Implementation sketch:

1. Parser: replay keyword followed by a quoted or bare path. The path lexing might need a small new helper (current parser doesn't have a "file path" terminal).
2. Command AST: Command::Replay { path: String }.
3. Runtime: read file, iterate lines, parse-and-execute each, abort on first failure. Probably best kept transactional at the file level (no individual command commits if any later one fails) — but that's a design question worth flagging in the implementation. Default to "stop on first error, report line number, don't roll back": matches the "I'm replaying my history" mental model where partial replay is a recoverable state.
4. AppEvent + handler for replay outcome.
5. Tests: happy path (3-line replay), failure-mid-replay (reports line number + stops), empty file, blank lines skipped, comment lines (# ...) skipped.

Watch out for: ADR-0015's history.log format — entries are append-only DSL command lines. replay history.log on a project should reproduce its current state if started from an empty database. That's the implicit invariant the test suite should prove.

Estimated: 3–4 hours.

B1. Update help text for ADR-0017 + ADR-0018 features

Scope: the in-app help command's output (in app.rs, the do_help or similar function around line 1100–1200) shows DSL command shapes. ADR-0017 added --force- conversion and --dont-convert flags (already added to help). ADR-0018 changed semantics of add column ... (serial|shortid) on non-empty tables (now auto-fills existing rows + emits UNIQUE) — this isn't called out anywhere user-facing.

Why independent: the ADRs spell out the behaviour; the help text just needs to surface it.

Suggested additions:

• add column ... (serial|shortid) line gains a sub-line: (existing rows auto-filled with sequence/generated values).
• change column ... (serial|shortid) similarly.
• New section "Auto-generated types" explaining serial and shortid in 3-4 lines.

Done when: the help output describes the behaviour matching ADR-0018 + ADR-0017. Existing help-output tests pass (some may need string-matching updates).

Estimated: 30 min.

B2. Test gap: change_column → bool from int 0/1

Scope: the type_change matrix has (Int, Bool) per- cell-classified (clean for 0/1, incompatible otherwise). This is well-tested at the matrix unit-test level. But there's no end-to-end test in db.rs exercising change column T: x (bool) from an int column. Trivial coverage gap to fill.

Why independent: identical pattern to existing change- column tests; just a different type pair.

Suggested test:

• change_column_type_int_to_bool_with_zero_one_succeeds: rows with values 0, 1, 0 → success, no client-side note expected (storage class doesn't change).
• change_column_type_int_to_bool_refuses_other_values: row with value 2 → incompatible refusal.

Done when: 2 new tests pass; total 536.

Estimated: 30 min.

Sharp edges and subtleties (delta vs. handoff-4)

Carried-over edges still apply (sync update, worker thread, metadata transactions, rebuild-table primitive, modal infrastructure, project-switch lock dance, [temp] cleanup guards, persistence ordering, DataResult carries column_types, output_render is the only place tabular output should originate). New ones this session:

• Type::Serial no longer implies PK at the type system level. ADR-0018 generalised serial. Existing references to "serial" in code comments may say "PK type" — those are stale. The non-PK serial path is active and tested.

• add column returns AddColumnResult, not TableDescription. Tests that called db.add_column(...).await.unwrap() and used the result as a description directly need .description indirection. Five existing tests were updated; new tests should follow the new shape.

• ChangeColumnTypeResult.client_side is now Option<ClientSideNote> where ClientSideNote carries transformed, lossy, auto_filled, auto_fill_kind. When auto-fill happens (target is serial/shortid + null cells), the note fires even though transformed is 0. The filter note.transformed > 0 || note.auto_filled > 0 is the canonical "should we emit a note" test.

• Non-PK serial INSERT auto-fill happens via MAX(col)+1. Per ADR-0010, the worker-thread serialisation makes this safe without explicit locking. If you ever extract the worker thread or change the connection model, this is one of the things that breaks.

• schema_to_ddl emits inline UNIQUE for non-PK columns flagged unique. PK columns aren't separately marked unique in ReadColumn (PK already implies it); the schema_to_ddl filter unique && !primary_key matters.

• read_schema reads UNIQUE via pragma_index_list filtered to origin = 'u'. Compound UNIQUE constraints are deliberately ignored (ADR-0018 OOS-6 / future C3). If you ever add multi-column UNIQUE support, the detection logic needs extending.

• Parse-error messages now show grammar-derived expected/found and a consumed-context prefix. Existing tests that asserted on the old message shape may have needed updates — none did, since the structural-error tests assert on substrings (the consumed context, the expected token).

Repository layout (delta vs. handoff-4)

src/
  type_change.rs               — new (ADR-0017)
  db.rs                        — many additions:
                                  AddColumnResult, ChangeColumn­
                                  TypeResult, ClientSideNote,
                                  AutoFillKind, ReadColumn.unique,
                                  read_unique_columns,
                                  schema_to_ddl UNIQUE emission,
                                  do_add_plain_column / do_add_auto_
                                  generated_column,
                                  do_change_column_type rewrite,
                                  run_change_column_with_dry_run +
                                  fill_auto_generated_cells,
                                  generate_shortid_batch,
                                  format_auto_fill_add_note,
                                  diagnostic helpers (lossy /
                                  incompatible / collision)
  dsl/
    parser.rs                  — change_column flag parsing,
                                  RichPattern-aware humanise,
                                  identifier .labelled,
                                  consumed-context rendering
    command.rs                 — ChangeColumnMode enum
    value.rs                   — validate_date / validate_datetime
                                  made pub(crate) so type_change
                                  can consume them
  app.rs                       — handle_dsl_change_column_success,
                                  handle_dsl_add_column_success,
                                  source-echo + caret on parse fail
  event.rs                     — DslChangeColumnSucceeded,
                                  DslAddColumnSucceeded
  output_render.rs             — render_diagnostic_table public,
                                  Alignment public,
                                  numeric_alignment_for public
  persistence/
    mod.rs                     — ColumnSchema.unique
    yaml.rs                    — write_column emits unique flag,
                                  RawColumn parses it
    csv_io.rs                  — test fixture updated
  runtime.rs                   — CommandOutcome::ChangeColumn
                                  + AddColumn variants
docs/
  adr/
    0017-column-type-change-compatibility.md
                               — §3 (serial→int row), §7 (PK
                                 identifiers) amended
    0018-auto-fill-contracts-for-serial-and-shortid.md
                               — new (this session)
    README.md                  — indexed
  handoff/
    20260508-handoff-5.md      — this file

How to take over

1. Read this file.
2. Read CLAUDE.md for the working-style rules.
3. Read docs/requirements.md for granular progress.
4. If picking up an Independent work item (§A1–B2): read just that item plus the listed ADR section it refers to. The items are scoped to be independently tackleable.
5. If working on H1 / Query DSL / Parser-as-source-of- truth: start with an ADR draft. Don't implement without one — those touch enough code to warrant the discipline.
6. Run cargo test to confirm the 534-test green baseline.
7. cargo clippy --all-targets to confirm clippy-clean.
8. cargo run --release to see the UI.

End-to-end smoke test (current state)

Demonstrates ADR-0017 + ADR-0018 features. Replaces the handoff-4 recipe (which is now stale — change column under ADR-0017 emits [client-side] notes the previous recipe didn't show).

$ rm -rf /tmp/handoff5-smoke
$ rdbms-playground --data-dir /tmp/handoff5-smoke

# Inside the app:
help                                       -- help text
                                              (B1: extend with
                                              ADR-0018 wording)
create table Customers with pk id:serial
add column Customers: Name (text)
add column Customers: Score (int)
insert into Customers ('Alice', 10)
insert into Customers ('Bob', 20)
insert into Customers ('Carol', 30)
show data Customers                        -- pretty-table render

# ADR-0017 type-change with [client-side] note:
change column Customers: Score (real)
                                           -- emits:
                                           -- [client-side] 3 row(s)
                                           -- were transformed before
                                           -- being stored. ...

# ADR-0017 lossy refusal:
change column Customers: Score (int)
                                           -- emits a bordered
                                           -- diagnostic table
                                           -- listing the lossy rows
                                           -- by PK; suggests
                                           -- --force-conversion.

change column Customers: Score (int) --force-conversion
                                           -- succeeds with both
                                           -- "transformed" and
                                           -- "lossy" counts in note.

# ADR-0018 add column auto-fill:
add column Customers: Tag (shortid)        -- emits:
                                           -- [client-side] 3 row(s)
                                           -- given auto-generated
                                           -- shortid values. ...
show data Customers                        -- Tag column populated

# ADR-0018 non-PK serial INSERT auto-fill:
add column Customers: Seq (serial)         -- emits another
                                           -- [client-side] note
insert into Customers ('Dave', 40)         -- Seq auto-fills 4
                                           -- (MAX of existing
                                           -- 1,2,3 plus 1)

# ADR-0018 int -> serial round-trip:
add column Customers: Counter (int)
update Customers set Counter=1 where id=1
update Customers set Counter=2 where id=2
update Customers set Counter=3 where id=3
update Customers set Counter=4 where id=4
change column Customers: Counter (serial)
                                           -- succeeds (no auto-fill
                                           -- needed since values
                                           -- are unique non-null)

# ADR-0017 PK FK-cascade refinement:
add column Customers: Email (text)
update Customers set Email='alice@example.com' where id=1
update Customers set Email='bob@example.com' where id=2
update Customers set Email='carol@example.com' where id=3
update Customers set Email='dave@example.com' where id=4
change column Customers: id (int)          -- serial -> int on PK,
                                           -- no inbound FK ->
                                           -- allowed.
change column Customers: id (serial)       -- int -> serial round
                                           -- trip succeeds.

# Parser tiny-win demo:
change column Tag in Customers: Tag (text)
                                           -- typo: column-name-
                                           -- first. Error now reads
                                           -- "after `change column
                                           -- Tag`, expected `:`,
                                           -- found `in`" with caret
                                           -- under the offending
                                           -- character.

quit

Manual spot-checks worth running

• --help lists all column ops (drop / rename / change) with their flags.
• Pretty rendering kicks in for show data AND every schema-mutating command's auto-show.
• change column T: c (real) succeeds and emits the [client-side] note for any non-empty table where the source values differ in storage class from the target.
• change column T: c (real) --force-conversion accepts fractional → int truncation; the note carries both counts.
• change column T: c (real) --dont-convert bypasses the client-side layer entirely (no [client-side] note, even if all cells transformed cleanly).
• add column T: x (shortid) on a non-empty table fills every existing row's x with a generated shortid.
• add column T: x (serial) on a non-empty table fills with 1..N. Subsequent inserts get N+1, N+2…
• Non-PK serial UNIQUE: update T set Seq=1 --all-rows → engine refuses with a unique-violation diagnostic.
• Save/load round-trip: create a non-PK serial column, quit, re-open. Read back: column is still UNIQUE.
• change column id (int) on a serial PK with no inbound FKs → allowed (per ADR-0017 §4.1 refinement).
• change column id (text) on a serial PK with an inbound FK → refused (per ADR-0017 §4.1 — fk_target_type would change).