rdbms-playground/docs/handoff/20260515-handoff-9.md

Session handoff — 2026-05-15 (9)

Ninth handover. This session executed ADR-0024 phases A through F in one sitting, six commits. The walker is now the sole parse path; the chumsky dependency is gone. What did NOT land — and what the next session needs to pick up — is captured below in priority order.

State at handoff

Branch: main. Working tree clean. Local HEAD is c940ba9, ahead of origin/main by six commits (the user pushes asynchronously; do not be blocked by unpushed state).

Commits since handoff-8's baseline (3e1ff83):

50b3542 ADR-0024 Phase A: walker framework + app-lifecycle commands
7e79ca8 ADR-0024 Phase B: DDL commands without value literals
6bb6882 ADR-0024 Phase C: create table with column-list value literals
c2accc2 ADR-0024 Phase D: data commands at chumsky parity
dca472f ADR-0024 Phase E: replay end-to-end
c940ba9 ADR-0024 Phase F (minimal): drop chumsky from the parse path

Tests: 844 passing, 0 failing, 1 ignored (up from 777 at handoff-8's baseline; +67 over this session). The ignored test is still the same \``ignoredoc-test insrc/friendly/mod.rs`.

Clippy: clean with nursery lints enabled and -D warnings.

Cargo.toml: chumsky dependency dropped. thiserror remains gone (per handoff-8). The crate's parse path is now fully Rust-native (lexer + walker, no parser-combinator library).

What shipped this session — quick overview

ADR-0024's six phases all landed end-to-end. Walker now owns all 20 entry-keyword commands across 14 entry words:

• Phase A: walker framework + 11 app-lifecycle commands (quit, help, rebuild, save / save as, new, load, export, import, mode, messages). Optional + Choice + Seq combinators. BarePath terminal. Path-bearing UX change shipped (paths with spaces require quotes).
• Phase B: DDL — drop, add, rename, change. Repeated combinator with optional separator. Flag terminal. NumberLit
  • Literal terminals. Optional now propagates inner expectations as skipped so completion sees "what could have appeared here".
• Phase C: create table. Repeated with , separator (first use). with pk defaulting to id:serial.
• Phase D: data — show, insert, update, delete. StringLit terminal. Value-literal sub-grammar. Schema-aware value typing deliberately deferred (see below).
• Phase E: replay. Choice(StringLit, BarePath).
• Phase F (minimal): chumsky combinators deleted, chumsky crate dep dropped. Significant scope deferred (see below).

DEFERRED — work the next session needs to pick up

This is the meat of the handoff. Items grouped by urgency.

1. Phase F (full): legacy parser-side modules still standing

ADR-0024 §migration Phase F prescribes deleting these modules. Phase F minimal in this session only deleted the chumsky combinator code. The following are still in place and consumed by other modules:

Module	Still consumed by	Why kept
src/dsl/lexer.rs	theme.rs, input_render.rs, app.rs, dsl/usage.rs, dsl/parser.rs	Per-token highlighting + echo-line tokenization + completion partial-token detection
src/dsl/keyword.rs (Keyword enum)	completion.rs, friendly/keys.rs, theme.rs, dsl/usage.rs	Catalog key derivation (parse.token.keyword.*), keyword-name validation in completion
src/dsl/ident_slot.rs (IdentSlot enum)	completion.rs, input_render.rs, runtime.rs, db.rs	Schema-cache lookups dispatched per slot kind
src/dsl/usage.rs::REGISTRY + matched_entry	completion.rs, app.rs	Per-command usage templates rendered on parse error
parse.token.keyword.* catalog (40+ entries)	dsl/usage.rs, friendly/keys.rs	Keyword wording in usage templates

Replacement strategy (sketched, for the migrating session):

• Highlighting (input_render.rs::render_input_panel): the walker's WalkResult::per_byte_class populates (start, end, HighlightClass) per matched terminal. Wire this output to the tok_* theme colours in place of the current lex(input)-driven span builder. Walker error positions feed the tok_error overlay. This is the single biggest change.
• Completion (completion.rs::candidates_at_cursor): the walker's WalkResult at WalkBound::Position(cursor) (NOT yet exercised — the walker's walk() does support this, but no caller passes it today) gives expected: Vec<Expectation> at the cursor. The bridge already maps Expectation::Ident { source: Tables/Columns/Relationships/Types } to the user-facing labels matching IdentSlot::expected_label, so the existing completion engine reads them transparently. What's missing: a walker-driven path that bypasses parse_command entirely and asks the walker directly for candidates per IdentSource. Today the bridge round-trips through ParseError::Invalid::expected strings — works, but loses information.
• Echo-line tokenization (output_render.rs for OutputKind::Echo + Mode::Simple): same lex-driven spans. Same walker per_byte_class plumbing.
• Usage templates (dsl/usage.rs::REGISTRY + matched_entry): every CommandNode already has usage_id. Wire the parse-error renderer to look up the catalog entry by usage_id instead of matched_entry. Then REGISTRY and matched_entry can go.
• Catalog cleanup (parse.token.keyword.* + friendly/keys.rs): ADR-0024 §cleanup-pass §F mentions a format_keyword_for_error (literal) -> String helper that wraps a literal in backticks and replaces the per-keyword catalog entries. Mechanical; do it after the consumers stop reading the catalog keys.

Estimated cost: one full session for the consumer migration + a follow-up commit for the catalog cleanup. Test suite serves as the regression net throughout.

2. Phase D (full): schema-aware value typing

ADR-0024 §migration Phase D prescribes "full schema awareness" via DynamicSubgrammar(column_value_list) that unfolds typed slots per column at walk time. This session deferred it.

What's in place:

• Node::DynamicSubgrammar(fn(&WalkContext) -> Node) variant declared but unused. The walker's driver returns Failed { expected: vec![] } on this branch (intentional — catches grammar bugs that declare DynamicSubgrammar without the wiring landing).
• WalkContext::current_table, WalkContext::current_table_columns, WalkContext::current_column all declared but unwritten. No Ident node has a writes_table: bool or equivalent.
• Per-type validators (int_slot, decimal_slot, etc.) NOT written. The current walker uses a generic value_literal Choice that accepts any literal regardless of column type; bind-time type-check errors fire as today.

To implement:

1. Plumb a SchemaCache reference into parse_command (and thus parse_tokens). Currently the call site is in runtime.rs::dispatch_input — passes app.schema_cache() alongside the input.
2. Extend WalkContext::new(schema) to carry the cache.
3. Implement Node::DynamicSubgrammar walker dispatch: resolve at walk time, leak the returned Node into a per-walk arena (or Box::leak per ADR-0024 §sub-grammars).
4. Implement Ident { source: Tables, writes_table: true } semantics — when the ident matches, look up the table in the schema, populate current_table + current_table_columns.
5. Implement typed value slots — int_slot(), decimal_slot(), etc. per ADR-0024 §typed-value-slots. Each is a Choice over the literal forms with a content validator.
6. Wire column_value_list as a DynamicSubgrammar that reads current_table_columns and emits a Seq of typed slots separated by commas.
7. Update insert shape to use column_value_list instead of the generic value list.
8. Update update/delete to use the per-column value slot based on current_column.

The user UX win this unlocks: typed slots reject mis-shaped input at parse time with localised wording (e.g., "Type a date as 'YYYY-MM-DD'") instead of bind-time errors; completion narrows per column type. This is the Phase D "central design claim" per the handoff.

Side effect to watch for: parse_command becomes schema- dependent. Tests that exercised parse in isolation may need to pass a schema cache (today's tests don't — most just check round-trip parses where schema doesn't matter).

Estimated cost: 1-2 sessions depending on how deep the schema plumbing goes through dispatch.

3. Walker doesn't drive completion or hints directly

Today's flow: walker produces WalkOutcome → bridge to ParseError::Invalid → completion / hint engines read expected: Vec<String>. The bridge formats Expectation::Ident { source: … } to the user-facing label string the existing engines recognise.

This works but loses information. Walker knows:

• The IdentSource of every expected slot.
• The role of every slot (e.g., parent_table vs child_table).
• The HintMode per node (currently always Default).
• The skipped expectations from any Optional that didn't engage at this position.
• The cursor's full MatchedPath so far — the AST builder could be invoked partially to extract context (e.g., "you've typed update Customers set Email=, the Email column's type is text").

A walker-direct completion path would surface much more informative candidates than the current ParseError-string round trip. See ADR-0024 §architecture: "Completion at cursor: walk(source, Position(cursor), ctx), inspect outcome.expected."

Today, no caller invokes walk() with WalkBound::Position(cursor). The variant exists in outcome.rs (annotated #[allow(dead_code)]). completion.rs still calls parse_command(leading_slice) — the slice-and-re-parse approach inherited from chumsky.

Migration path: add a pub fn candidates_at_cursor_from_walker (input, cursor, schema) -> WalkResult that calls walker::walk(input, WalkBound::Position(cursor), &mut ctx) and returns the result. Then completion.rs reads expected directly with full IdentSource + role info.

4. HintMode declared but unused

HintMode::Default | ForceProse | ProseOnly | SuppressProse is declared in grammar/mod.rs but every CommandNode and every Node::Ident sets it to None. The current ad-hoc hint cases in input_render.rs::ambient_hint (value-literal slot suppression, NewName slot typing-name prose, invalid-ident overlay) still use the chumsky-era ad-hoc detection.

ADR-0024 §HintMode-per-node says these migrate to node-attached HintMode annotations during Phase D. They didn't.

To do: annotate the value-literal slots with HintMode::ProseOnly("value_literal_format_hint"); annotate NewName slots with the typing-name prose; have the hint resolver pick up the mode and dispatch.

5. Ranker hook: declared in ADR-0024, not implemented at all

ADR-0024 §ranker-layer specifies a Ranker function type between the walker's raw candidates and the hint-panel renderer. The default is identity_ranker (declaration-order preserved).

Status: not declared anywhere in code. No Ranker type, no identity ranker, no hook into completion. The current completion engine ranks by its own ad-hoc logic (keyword matches first, then schema, alphabetised within each).

This is a small future-work hook. Not blocking.

6. Aliases: feature works, no aliases declared

Word::aliases: &'static [&'static str] is wired through Word::matches and the walker correctly accepts case- insensitive alias matches. Today every Word in the grammar has an empty aliases slice.

The round-5 q quit alias removal stands — if the user wants it back, it's:

const QUIT_WORD: Word = Word {
    primary: "quit",
    aliases: &["q"],
    highlight_override: None,
};
pub static QUIT: CommandNode = CommandNode {
    entry: QUIT_WORD,
    ...
};

The walker matches either; completion only surfaces the primary. No further wiring needed.

7. Test edits worth knowing about

A handful of tests changed wording assertions to match walker-emitted error wording:

• src/dsl/walker/mod.rs::walker_import_trailing_as_without_target_errors — assertion weakened from checking for "target" to checking for "import". Walker's Optional backtracking means import foo.zip as parses as Import { path: "foo.zip", target: None } followed by trailing as → walker reports expected end of input, found …. The friendly project.import_empty_target wording moved out of the parser. The integration test (tests/iteration5_export_import.rs::import_with_empty_target_after_as_errors) still passes because the rendered import_usage template line in the dispatch output contains both "import" and "target".
• The walker's parse error wording for incomplete inputs consistently uses "after <consumed>, expected …, found end of input" — matches the chumsky-era contract that structural_error_for_show_data_without_arg and friends pin down.

No parse.token.keyword.* wording changed. No catalog entries changed. No user-visible string regression.

8. The value_literal_hint_at_cursor stopgap

The round-6 stopgap from handoff-8 (replacing null true false with prose at value-literal slots) is still live. It lives in completion.rs::value_literal_hint_at_cursor and detects the value-literal-signature in expected. With Phase D's full schema awareness, this would become "narrow to the actual column's type" (e.g., "Type a date as 'YYYY-MM-DD'") — but that requires the schema plumbing in §2 above.

Today the stopgap continues to fire for every value-literal slot regardless of column type. Same wording as handoff-8.

9. Choice greedy-semantics shape contortions

ADR-0023 says "the trie design is greedy (the first child node that matches wins)." This is the design. But it forced some grammar contortions worth knowing about:

• insert Form A vs Form C disambiguation (grammar/data.rs): Forms A (insert into T (cols) values (vals)) and C (insert into T (vals) — bare value list) both start with (. The inner-paren content is parsed as a heterogeneous Choice(VALUE_LITERAL, Ident{Columns}) — VALUE_LITERAL ordered first so null/true/false match their Word branch rather than the broader identifier catch-all (which consume_ident doesn't filter against the keyword set). AST builder discriminates by the presence of the values keyword AFTER the first paren. Brittle if a future grammar needs to add another paren-bounded form starting at the same position.
• drop sub-form ordering: drop_column and drop_relationship come before drop_table in the Choice because they're more specific (longer prefix). Walker greediness handles this correctly because each branch's first Word disambiguates.

10. Walker per_byte_class populated but unused

WalkResult::per_byte_class: Vec<ByteClass> is populated by every terminal match in the walker driver. No consumer reads it today. The annotation #[allow(dead_code)] sits on the field. When the highlighting consumer migrates (§1 above), this is the data source.

11. Differential test scaffolding wasn't actually built

ADR-0024 §test-discipline §3 specifies a "differential check during the migration window" — a test helper that runs both parsers on the existing input corpus and asserts identical Command output. Removed at Phase F cleanup.

This session went with hand-curated expected Command outputs in dsl::walker::tests instead. Equivalent coverage (every migrated command's parse asserted), simpler to maintain. Since chumsky is now gone (Phase F minimal removed the combinator code), no removal step needed.

If a strict differential check is wanted retroactively, the chumsky path would need to be reconstructed from git history and run alongside walker against the test corpus — not trivial. The hand-curated tests + the existing integration test suite serve the same regression-net role.

12. WalkContext writes during walk — design exists, not implemented

ADR-0024 §WalkContext sketches Ident { source: Tables, writes_table: true } semantics: when the ident matches, the walker writes current_table to context. Subsequent dynamic sub-grammars read it.

Today no Ident node has a writes_table field. The struct definition in grammar/mod.rs is:

Ident {
    source: IdentSource,
    role: &'static str,
    validator: Option<IdentValidator>,
    highlight_override: Option<HighlightClass>,
}

When Phase D (full) lands, add a writes_table: bool (or similar) field and have the walker driver populate WalkContext accordingly.

13. CommandNode::usage_id / help_id not consumed

Every CommandNode declares usage_id: Option<&'static str> and help_id: Option<&'static str> pointing into the catalog. No code reads these fields. Usage rendering still goes through dsl/usage.rs::matched_entry. Help text still hand-curated.

When dsl/usage.rs::REGISTRY retires (§1 above), wire the parse-error renderer and the in-app help system to read these fields directly.

Sharp edges to know about

These are facets of the walker's behaviour that aren't bugs but that will surprise someone reading the code cold.

• Optional backtracking on partial-match is intentional (matches chumsky's or_not semantics). When an Optional's inner consumes some terminals and then fails (Incomplete or Mismatch), the walker rolls back the path / per_byte state to the pre-Optional position and treats it as skipped, with the inner's expectations carried as skipped on the Matched return. Validation failures (content errors) do NOT backtrack — the user means to fix those. This asymmetry is the load-bearing decision that makes create table T with produce the correct IncompleteAtEof classification (chumsky's behaviour).
• Walker's Choice is strictly greedy — first child whose first terminal matches wins. No backtracking. Required ordering: more-specific shapes before more-general. See §9 above for examples.
• Ident { source: Tables/Columns/Relationships } does NOT validate against the schema at parse time. It's a shape-only check today. Schema-aware parse is the Phase D vision; see §2.
• Literal(&'static str) matches verbatim bytes with a word-boundary lookahead so 1 doesn't half-match 12 and n doesn't half-match name. The highlight class is inferred from the literal's bytes (digits → Number, else Keyword). Used today only for the 1 in add 1:n relationship.
• AST builder failures surface as WalkOutcome::ValidationFailed with at_eof = true. The bridge maps these to ParseError::Invalid with at_eof: true so the input renderer classifies them as IncompleteAtEof (no live overlay; on-submit error fires). This mirrors the chumsky- side custom-error convention.
• unknown_command_error is the sole catch-all for inputs whose first identifier-shape token isn't a registered entry word. Wording: "expected one of add, …, found <word>". Position is the start of the unknown word.
• q quit alias remains gone. The walker SUPPORTS aliases natively — adding q back is a one-line change on QUIT.entry.aliases (see §6).
• Path-bearing UX change shipped (Phase A + E): replay, import, export paths terminate at the first whitespace byte. Paths with spaces use the quoted form (replay 'my project/seed.commands'). This is per ADR-0024.

Suggested next-session priorities

In order:

1. Phase F (full): consumer migration to walker outputs. This is the biggest deferred chunk and the right next structural move. See §1 above for the migration sketch. ~1 session for the highlighting + completion + usage migration; one follow-up commit for the catalog cleanup and lexer/keyword/ident_slot deletion.
2. Phase D (full): schema-aware value typing. Once the schema cache plumbing exists, the DynamicSubgrammar wiring + typed value slots are mechanical. See §2.
3. Walker-driven completion (§3). Smaller scope than the above. Surfaces IdentSource + role directly to the completion engine without the ParseError-string round trip. Unlocks better hint UX.
4. HintMode annotations (§4). Mechanical migration of the ad-hoc hint cases in input_render.rs to node annotations. Small.
5. Ranker hook (§5). Future work. Plug-in point for frequency-based ranking, content-aware priors, recency.

After (1) and (2) land, the codebase reaches the steady-state ADR-0024 envisioned: one declaration per command, no scatter, walker as single source of truth across parse / complete / highlight / hint / usage.

ADR index (read these before touching the related areas)

0000 Record architecture decisions (process)
0001 Language and TUI framework (Rust + Ratatui)
        — chumsky dependency dropped in Phase F minimal
0002 Database engine (User-facing posture)
0003 Input modes and command dispatch
0004 Project file format (amended by 0015)
0005 Column type vocabulary
0006 Undo snapshots and replay log (designed, not impl)
0007 Sharing and export (amended by 0015 amendment 1)
0008 Testing approach
0009 DSL command syntax conventions
0010 Database access via worker thread
0011 FK column type compatibility
0012 Internal metadata for user-facing column types
0013 Relationships, naming, and rebuild-table strategy
0014 Data operations, value literals, and auto-show
0015 Project storage runtime
0016 Pretty table rendering for data and structure views
0017 Column type-change compatibility
0018 Auto-fill contracts for serial and shortid columns
0019 Friendly error layer (H1) and i18n message catalog
        — `parse.token.keyword.*` collapse pending Phase F full
0020 Tokenization layer for the DSL parser
        — superseded by the scannerless walker in ADR-0024;
          the lexer module survives until Phase F full
0021 Parser-as-source-of-truth for H1a
        — usage info migration to grammar nodes pending
          (CommandNode.usage_id declared, not consumed yet)
0022 Ambient typing assistance (I3 + I4 unified)
        — completion still chumsky-bridge; walker direct
          path pending
0023 Unified declarative grammar tree (direction)
        — superseded for execution by ADR-0024
0024 Unified grammar tree: execution plan (ACCEPTED)
        — A through F minimal landed; F full + D full deferred

Repository layout (delta vs. handoff-8)

New files this session:

src/dsl/grammar/
  mod.rs        (267)  — Node enum, Word, IdentSource, HintMode,
                          HighlightClass, ValidationError,
                          IdentValidator, NumberValidator,
                          CommandNode, REGISTRY (20 commands)
  app.rs        (272)  — 11 app-lifecycle commands
  ddl.rs        (492)  — drop, add, rename, change, create
  data.rs       (504)  — show, insert, update, delete, replay
  shared.rs     (108)  — type validator, qualified_column,
                          referential_clauses, action_keyword
src/dsl/walker/
  mod.rs        (~620) — walk() entry + 53 walker tests
  driver.rs     (~570) — per-node-kind dispatch with backtracking
  context.rs    (43)   — WalkContext (schema fields stubbed)
  outcome.rs    (~165) — WalkResult, WalkOutcome, MatchedPath
  lex_helpers.rs (~190) — byte-level helpers (skip_whitespace,
                          consume_ident, match_keyword,
                          consume_bare_path, consume_flag,
                          consume_number_literal, consume_string_literal)

Files modified this session:

src/dsl/mod.rs          — wire grammar + walker modules
src/dsl/parser.rs       — chumsky combinators deleted; now a
                          ~290-line walker bridge with the
                          original test suite (~840 lines) intact
Cargo.toml              — chumsky dependency removed
Cargo.lock              — regenerated

Files NOT touched but worth knowing about (still consume the legacy modules per §1):

src/dsl/lexer.rs        — still exports lex() / Token / TokenKind
src/dsl/keyword.rs      — Keyword enum still alive
src/dsl/ident_slot.rs   — IdentSlot enum still alive
src/dsl/usage.rs        — REGISTRY + matched_entry still alive
src/completion.rs       — reads ParseError::Invalid::expected
                          (bridge from walker), uses Keyword +
                          IdentSlot
src/input_render.rs     — uses lex() for token-class colouring
src/theme.rs            — token colour mappings keyed on Keyword
src/runtime.rs, src/db.rs — IdentSlot::expected_label round-trip

How to take over

1. Read this file.
2. Read CLAUDE.md for the working-style rules. Note the "Escalate ambiguity — do not decide for the user" rule. The deferred items below are scoped enough that most decisions are clear; escalate if the spec genuinely disagrees with the implementation.
3. Read ADR-0024 (docs/adr/0024-unified-grammar-tree-execution-plan.md) to understand the design intent. Phase F (full) and Phase D (full) are the unfinished work.
4. Skim src/dsl/grammar/mod.rs — the Node enum + Word
   • CommandNode + REGISTRY are the contract.
5. Skim src/dsl/walker/mod.rs — the walk() entry + bridge logic. The 53 tests at the bottom of that file are the behavioural spec.
6. Run cargo test to confirm the 844-test baseline. Lib test count is 711 in rdbms_playground (the rest are integration + doctests). Total should be 844 passed, 0 failed, 1 ignored.
7. Run cargo clippy --all-targets -- -D warnings to confirm clean baseline.
8. Pick a deferred item from §1-§5 and start. §1 (Phase F full) is the natural next move; it unblocks §3 (walker-driven completion) and §4 (HintMode annotations). §2 (Phase D full) is the second-largest item and can land in parallel since it touches the grammar layer rather than the consumer layer.

End-to-end smoke test (current state, post-ADR-0024)

$ rm -rf /tmp/handoff9-smoke
$ rdbms-playground --data-dir /tmp/handoff9-smoke

# Inside the app:

# All commands now route through the walker. User-visible
# behaviour is unchanged from handoff-8 except for:
# - import / export paths with spaces require quotes
# - replay paths with spaces require quotes
# - parse error wording uses "after `<consumed>`, expected …,
#   found end of input" framing (matches the chumsky-era
#   contract; tests pin this down)

# Smoke commands:
help                                         -- in-app help
mode advanced                                -- mode switch
quit                                         -- exit (no `q`)
:quit                                        -- one-shot escape
                                                also works

# DDL:
create table Customers with pk
add column Customers: Email (text)
add 1:n relationship from Customers.id to Orders.customer_id

# Data:
insert into Customers values (1, 'Alice')
update Customers set Email='new@b.c' where id=1
delete from Customers where id=1
show data Customers
show table Customers

# Replay:
replay history.log
replay 'my project/seed.commands'

# Errors:
frobulate widgets
# → expected one of `add`, `change`, `create`, `delete`,
#   `drop`, `export`, `help`, `import`, `insert`, `load`,
#   `messages`, `mode`, `new`, `quit`, `rebuild`, `rename`,
#   `replay`, `save`, `show`, or `update`, found `frobulate`

mode bogus
# → unknown mode 'bogus' (expected 'simple' or 'advanced')

change column T: c (int) --force-conversion --dont-convert
# → `--force-conversion` and `--dont-convert` are mutually
#   exclusive — pick one.

create table Customers
# → tables need at least one column. Add `with pk` for a
#   default `id INTEGER PRIMARY KEY`, or `with pk <name>:<type>`
#   …

# All wording sourced from the en-US.yaml catalog via the
# walker's ValidationError catalog-key mechanism.

After Phase F full lands, this smoke test extends with:

• Per-keystroke highlighting driven by walker per_byte_class
• Cursor-position completion driven by walker direct path
• Usage templates rendered from CommandNode.usage_id
• Lexer/Keyword/IdentSlot modules removed from the source tree