rdbms-playground/docs/handoff/20260508-handoff-4.md

# Session handoff — 2026-05-08 (4)

Fourth handover on what's been a busy day. The previous
session shipped track 2's export/import + --resume +
persistent history + migration scaffold (handoff-3). This
session built on top: pretty-table rendering, B2/C2 column
operations, a small UX polish pass, and an ADR-driven design
for column type-change compatibility (ADR-0017 — the next
session's main job to implement).

## State at handoff

**Branch:** `main`. Working tree clean. Commits since
handoff-3:

```
c3e5f90 ADR-0017 + ADR-0002 amendment: type-change compatibility
        + engine-agnostic posture
7b97786 B2/C2: column drop / rename / change-type DSL commands
41cef53 parser: make `to` and `table` independently optional
        in add column
1b27a0c runtime: suppress silent-rebuild banner for empty
        projects
5b5e08d ADR-0016 + Iter 5/6 follow-up: pretty table rendering
67d68db Iteration 6: --resume + persistent input history +
        migration scaffold
c6cf3df Iteration 5: export / import commands
```

**Tests:** **449 passing, 0 failing, 0 skipped** (up from
345 at handoff-2's baseline; +104 over the last two
sessions).

**Clippy:** clean with `nursery` lints enabled.

**Release build:** ~7.0 MB single binary (up ~600 KB from
handoff-3's 6.9 MB; the increase is the column-ops code
path and additional tests-only items don't ship). The zip
crate from Iter 5 is unchanged.

**Branch state vs origin:** 8 commits ahead of `origin/main`.
Push remains the user's call.

## What's implemented (delta vs. handoff-3)

The previous handoff covered: Iter 1–6 of track 2
(file-backed projects, per-command persistence, rebuild,
save/save as/new/load, export/import, --resume, persistent
history, migration scaffold). This session adds:

### Pretty table rendering (ADR-0016, commit 5b5e08d)

- New `src/output_render.rs` — hand-rolled,
  Unicode-box-drawing renderer with header-only border
  style (`┌─┐│└─┘├─┤`). Public surface:
  - `render_data_table(&DataResult) -> Vec<String>`
  - `render_structure(&TableDescription) -> Vec<String>`
- **Type-aware alignment**: `DataResult` was extended with
  `column_types: Vec<Option<Type>>`, populated from the
  same metadata lookup `describe_table` already does.
  Numeric types (int, real, decimal, serial)
  right-align; everything else left-aligns. No
  string-heuristics: types are the source of truth.
- **NULL rendering**: `(null)`. Cell newlines / tabs /
  control chars become `↵` / `→` / `·` for display only;
  underlying data is untouched and round-trips through
  CSV cleanly.
- 18 new unit tests in `output_render.rs` plus 4 insta
  snapshots pinning the canonical layouts.
- Replaces the old pipe-and-dash `render_data_view` in
  `app.rs`. The structure rendering now lives entirely in
  the renderer module; `handle_dsl_success` just calls it.

### Silent-rebuild banner fix (commit 1b27a0c)

A fresh-launch temp project hits the on-startup
"rebuild from text" path (because no `.db` exists yet)
and used to surface "[ok] rebuild — 0 tables and 0 rows
will be reconstructed." Now `runtime::run()` suppresses
the banner when the project has no schema. The explicit
`rebuild` command still always reports — the user
explicitly asked.

### Parser: optional `to`/`table` in add column (commit 41cef53)

Both prepositions are now independently optional, so all
four variants parse identically:

```
add column to table T: c (text)
add column to T: c (text)
add column table T: c (text)
add column T: c (text)
```

Matches the convention elsewhere in the DSL where bare
identifiers are accepted in unambiguous positions. New
helper `optional_keyword(kw)` that the column-ops
parsers (next bullet) reuse.

### B2/C2: drop / rename / change column commands (commit 7b97786)

Closes B2 (rebuild-table primitive reused outside
relationships) and C2 (full add/drop/rename/change-type
column operations). Three new DSL commands:

- **`drop column [from] [table] <T>: <col>`** — uses
  SQLite's native `ALTER TABLE … DROP COLUMN` (3.35+) +
  metadata cleanup in `__rdbms_playground_columns`.
  Refuses PK columns and columns involved in any
  declared relationship (drop the relationship first).
- **`rename column [in] [table] <T>: <old> to <new>`** —
  uses `ALTER TABLE … RENAME COLUMN` (3.25+); the engine
  cascades the rename through FK declarations on other
  tables. Mirrors the rename into BOTH metadata tables
  (`__rdbms_playground_columns` and
  `__rdbms_playground_relationships`) so describes stay
  accurate. Refuses identity rename and name collisions.
- **`change column [in] [table] <T>: <col> (<newtype>)`**
  — routes through the rebuild-table primitive
  (ADR-0013) since `ALTER TABLE` doesn't support type
  changes. **Currently** refuses PK columns,
  relationship-involved columns, `serial` target, and
  no-op same-type. Data conversion compatibility is
  delegated to STRICT typing during the
  `INSERT INTO new SELECT FROM old` step — which is the
  spec gap that ADR-0017 fills (next session).

20 new tests (11 parser, 9 db-layer) covering happy
paths, refusals, and metadata propagation.

### ADR-0017: column type-change compatibility (commit c3e5f90)

The big design piece of this session. The B2/C2 change
command's "let STRICT decide" placeholder is replaced
with a curated model. Headlines:

- **Per-cell three-tier outcome model**: every cell
  classifies as **clean** (transformer produces a value
  the target accepts losslessly), **lossy** (valid
  output, but information discarded — e.g. real → int
  truncation), or **incompatible** (no transformer can
  produce a valid output — e.g. text "abc" → int).
- **Static transformer matrix** for numeric chains,
  text↔structured types, and always-clean
  stringifications. The text → int chain narrowest-first:
  try int parse (clean), fall back to real parse + truncate
  (lossy), else incompatible.
- **PK columns refined**: B2/C2's blanket PK refusal
  becomes conditional. The cascade only fires when the
  new type would change `fk_target_type()` (per
  ADR-0011). So `serial → int` and `shortid → text` on
  FK-referenced PKs are now **allowed** (both preserve
  the FK target type); `int → text` etc. still refuse.
- **Uniqueness-bearing columns** (PK + shortid) get a
  post-transformation collision check. Lossy
  transformations that collapse distinct source values
  to the same target are classified as incompatible
  (cross-row, structural; no `--force-conversion`
  override).
- **Override flags**: `--force-conversion` accepts loss
  but still refuses incompatibles; `--dont-convert`
  skips the entire client-side layer (raw values to
  STRICT). Mutually exclusive.
- **Pedagogical `[client-side] …` note** in the success
  summary whenever the transformer rewrote any cell —
  the moment that tells the learner "the tool did this
  for you; raw SQL would need a CAST or application
  code."
- **Diagnostic row lists rendered via the pretty-table
  renderer** (ADR-0016) — bordered, capped at 100 rows,
  with the trailing `… and N more` row inside the box.
- **Forward-look** captured for `--default <value>` /
  `--on-incompatible '<value>'` resolution flags
  (preserves design space; not in v1).

Companion change: ADR-0002 gained a new "User-facing
posture" section cementing that the engine choice is an
implementation detail and is never named in user-visible
strings (no "SQLite", no "STRICT", no "PRAGMA"). A
matching bullet was added to `CLAUDE.md`'s working-style
rules so every session picks it up.

## ADR index (read these before touching the related areas)

```
0000 Record architecture decisions (process)
0001 Language and TUI framework (Rust + Ratatui)
0002 Database engine
        — amended in this session: User-facing posture
          (no engine name in user-visible strings).
0003 Input modes and command dispatch
0004 Project file format
        — amended by 0015
0005 Column type vocabulary
0006 Undo snapshots and replay log (deferred)
0007 Sharing and export
        — amended by 0015 amendment 1 (history.log out of
          export zip)
0008 Testing approach
0009 DSL command syntax conventions
0010 Database access via worker thread
0011 FK column type compatibility
        — referenced by 0017 §4.1
0012 Internal metadata for user-facing column types
0013 Relationships, naming, and rebuild-table strategy
        — primitive reused by ADR-0017's change-column
          implementation
0014 Data operations, value literals, and auto-show
0015 Project storage runtime (track 2; fully implemented
     through Iterations 1–6)
0016 Pretty table rendering for data and structure views
0017 Column type-change compatibility
        — DESIGNED. Implementation pending; the next
          session's main task.
```

## Pending — proposed next moves (in order)

### 1. Implement ADR-0017

This is the biggest chunk and the obvious next step. The
ADR is fully specified — no further design work is needed
before coding. Estimated scope: ~600–800 lines + tests.

The work splits naturally into a few pieces:

1. **Transformer matrix module** (`src/dsl/types.rs`
   alongside `fk_target_type()`, or a new
   `src/type_change.rs`). Per-pair functions
   `transform(src_value, target_type) -> CellOutcome`
   where `CellOutcome ∈ {Clean(new), Lossy(new, reason),
   Incompatible(reason)}`. Reuse the validators in
   `dsl/value.rs` for grammar checks; the lossy/clean
   distinction is computed fresh per pair.
2. **Parser additions**: `--force-conversion` and
   `--dont-convert` flags on the `change column` parser.
   The flags conform to ADR-0009's `--<name>` convention.
   Mutual-exclusion check rejects both at parse time.
3. **`do_change_column_type` rewrite** in `db.rs`. Replace
   the current "refuse PK / refuse relationship-involved
   / call rebuild_table" body with:
   - Static refusal preconditions (incl. the refined
     `fk_target_type`-aware PK rule per ADR-0017 §4).
   - Per-cell dry run that reads each row's value and
     classifies via the matrix.
   - Post-transformation uniqueness check for PK +
     shortid columns.
   - If `--dont-convert`: skip dry run, hand raw to
     `rebuild_table` (engine STRICT enforces). Wrap any
     engine error in a friendly form (no engine name —
     CLAUDE.md rule).
   - Otherwise: refuse on incompatibles, refuse on lossy
     unless `--force-conversion`, proceed with rebuild
     using a new `rebuild_table_with_transform` variant
     (or a parameter on `rebuild_table`) that streams
     rows through the transformer rather than the
     identity SQL `SELECT`.
4. **Diagnostic rendering**: a small helper in
   `output_render.rs` (or alongside it) that produces
   the bordered tables for lossy / incompatible /
   uniqueness-collision cases per ADR-0017 §7.
5. **`[client-side]` note**: the success path adds the
   note when any cell was non-identity transformed.
6. **Tests**: one per cell-classification per pair
   (clean / lossy / incompatible), the FK-cascade
   refinement, the uniqueness check (PK + shortid),
   `--force-conversion` allowing lossy, `--dont-convert`
   bypassing client-side, mutual exclusion of the two
   flags. Probably 30–50 new tests.

The B2/C2 implementation lives in `db.rs` at
`do_change_column_type` (line ~1550 area). That's the
function to rewrite. The B2/C2 tests at `change_column_*`
in `db.rs::tests` will mostly stay relevant; some
assertions about the PK refusal will need updating to
match the new finer-grained rule (PK without inbound FK
is now allowed).

### 2. Query DSL ADR + implementation

The biggest remaining design piece. Earlier discussion
landed on: extend `show data` into a SELECT-style command
with WHERE / projection / order; expose generated SQL as a
pedagogical hook; bundle C5a's complex WHERE into one
coherent feature. Needs an ADR (operator subset, NULL
semantics, projection grammar, "see the SQL" UX). Then
QA1 (EXPLAIN QUERY PLAN) becomes meaningful since queries
will exist.

### 3. CI (TT5)

Test infrastructure exists; the GitHub Actions workflow
file does not. One file, locks in the green baseline
across Linux / macOS / Windows.

### 4. Friendly error layer (H1)

Partial — FK errors are enriched both ways. The remaining
work is a general engine-error → learner-language
translator. Per ADR-0002's new User-facing posture, the
posture is "never expose engine error text verbatim";
H1's full implementation is what makes that
operationally true.

### 5. `replay` command (U4)

`history.log` is already replay-compatible. The command
runs commands from a `history.log` or `.commands` file.
Modest scope.

### 6. Bigger UX projects

V4 (session log + Markdown export), V1/V2 pretty
rendering refinements (relationship-rendering ADR — the
"two structures + arrow" view), B2 column operations'
edge cases (e.g. UNIQUE constraints when C3 lands).

## Sharp edges and subtleties (delta vs. handoff-3)

Carried-over edges still apply (sync `update`, worker
thread, metadata transactions, rebuild-table primitive,
modal infrastructure, project-switch lock dance, the
`[temp]` cleanup guards, persistence ordering). New ones
this session:

- **`DataResult` now carries `column_types`.** Tests that
  construct `DataResult` directly need the field
  populated — `column_types: vec![Some(Type::…), …]`
  with one entry per column. Two existing tests in
  `walking_skeleton.rs` were updated for this; future
  tests follow the same shape.

- **The pretty-table renderer is `output_render.rs`, not
  `ui.rs`.** `ui.rs` is the ratatui Frame renderer for
  the whole TUI layout; `output_render` produces text
  rows (one `OutputLine`-equivalent per element) for
  the output panel. Different concerns; same project.

- **No engine name in user-facing strings.** `CLAUDE.md`
  has a working-style bullet codifying this (per
  ADR-0002's new "User-facing posture" section). When
  ADR-0017's implementation produces error messages,
  do not say "SQLite", "STRICT", "PRAGMA", etc. Say
  "the database" or describe the issue in the abstract.
  Existing code mostly already follows this; the
  occasional drift (e.g. `do_drop_table` may have a
  legacy mention — check) should be cleaned up
  opportunistically.

- **B2/C2's PK refusal is about to relax.** When you
  implement ADR-0017, the current "PK refused" tests
  for change-column-type will need updating: the rule
  becomes "refused only when the column has an inbound
  FK and the new type changes `fk_target_type`". The
  matching `*_refuses_pk` tests should split into a
  pair: "refused when FK-referenced and target-type
  changes" and "allowed when FK-referenced and target
  type preserved (`serial → int` happy path)".

- **Modern SQLite versions for column ops.** B2/C2
  uses `ALTER TABLE DROP COLUMN` (3.35+) and
  `ALTER TABLE RENAME COLUMN` (3.25+). The `rusqlite`
  bundled feature pins a recent SQLite, so these are
  available. If we ever switch to system-linked SQLite
  on a host without these, drop_column / rename_column
  would refuse — worth a follow-up if it happens.

- **`output_render.rs` is the only place new tabular
  output should be produced from now on.** Per
  ADR-0017 §7's directive (and per the user's
  "everything tabular goes through it" rule), any
  future feature that lists rows in error / success
  output uses `render_data_table` rather than
  hand-rolled indented lines.

## Repository layout (delta vs. handoff-3)

```
src/
  output_render.rs             — new (ADR-0016)
  dsl/
    parser.rs                  — drop_column / rename_column /
                                  change_column parsers,
                                  optional_keyword helper
    command.rs                 — DropColumn / RenameColumn /
                                  ChangeColumnType variants
  db.rs                        — do_drop_column,
                                  do_rename_column,
                                  do_change_column_type
                                  + Database::* sibling methods
                                  + DataResult.column_types field
  app.rs                       — calls into output_render;
                                  the inline render_data_view
                                  helper + join_padded +
                                  separator_row removed
  runtime.rs                   — DropColumn / RenameColumn /
                                  ChangeColumnType dispatch in
                                  execute_command_typed;
                                  silent-rebuild suppression
  snapshots/                   — 4 new insta snapshots
                                  (output_render renderings)
docs/
  adr/
    0002-database-engine.md    — User-facing posture amendment
    0016-pretty-table-rendering.md  — new (this session)
    0017-column-type-change-compatibility.md  — new (design only)
    README.md                  — indexed
  handoff/
    20260508-handoff-4.md      — this file
CLAUDE.md                      — "no engine name" working-
                                  style bullet
```

## How to take over

1. Read this file.
2. Read `CLAUDE.md` for the working-style rules and current
   layout.
3. Read `docs/requirements.md` for granular progress.
4. **Read ADR-0017 in full** — that's the spec for the
   first task. Also skim ADR-0016 (pretty-table renderer)
   and ADR-0002 (User-facing posture) since both inform
   the implementation.
5. Run `cargo test` to confirm the 449-test green baseline.
6. `cargo run --release` to see the UI; try a `change
   column` against the current B2/C2 implementation so
   you have a concrete reference for what changes when
   ADR-0017 lands.

### End-to-end smoke test (current state)

Demonstrates the new column ops + pretty rendering. Does
NOT exercise ADR-0017 (which is unimplemented):

```
$ rm -rf /tmp/handoff4-smoke
$ rdbms-playground --data-dir /tmp/handoff4-smoke

# Inside the app:
help                                    -- new column ops listed
create table Customers with pk id:serial
add column Customers: Name (text)       -- bare-identifier form
                                        -- (handoff-3 → handoff-4 polish)
add column to Customers: Email (text)   -- with `to`, no `table`
add column table Customers: Score (int) -- with `table`, no `to`
insert into Customers ('Alice', 'a@b.com', 10)
insert into Customers ('Bob', 'b@b.com', 20)
show data Customers                     -- rendered via pretty table:
                                        -- ┌────┬───────┬─────────┬───────┐
                                        -- │ id │ Name  │ Email   │ Score │
                                        -- ...

# Column ops:
rename column Customers: Score to Points
drop column Customers: Email
show table Customers                    -- pretty structure listing

# Type change (B2/C2 placeholder — works for safe cases,
# refuses for PK / relationships / serial / no-op):
change column Customers: Points (real)  -- succeeds (per-cell
                                        -- compatible: int → real
                                        -- widens cleanly)
quit
```

After ADR-0017 lands, the same `change column` will
produce client-side notes for transformed rows, refuse
lossy without `--force-conversion`, render bordered
diagnostic tables for refusals, etc.

### Manual spot-checks worth running

- `--help` lists all column ops (drop / rename / change).
- Pretty rendering kicks in for `show data` AND every
  schema-mutating command's auto-show.
- `[ok] rebuild — …` no longer fires on a fresh-launch
  empty temp project (verify by launching with no args).
- `add column [to] [table] T: c (type)` — try all four
  shapes; all parse.
- `change column T: id (text)` on a serial PK refuses
  (current B2/C2 behavior; will become conditional under
  ADR-0017 — refused only if FK-referenced AND
  fk_target_type changes).