rdbms-playground/docs/handoff/20260519-handoff-25.md

Session handoff — 2026-05-19 (25)

Twenty-fifth handover. Implementation session. ADR-0030 Phase 1 ("Foundations + first SELECT") and ADR-0031 (the SQL expression grammar) landed end to end: a SQL select typed in advanced mode parses, runs against the database, and renders through the existing data-table renderer; the same line in simple mode lights up the precise "this is SQL" hint instead of running, and Tab completion / live overlay / [ERR] indicator agree.

§1. State at handoff

Branch: main. Working tree clean.

Tests: 1260 passing, 0 failing, 1 ignored (the unchanged doctest). Up from the handoff-24 baseline of 1240 by +20 — 13 sql-expression unit tests, 7 SQL-SELECT integration tests. Clippy: clean under -D warnings.

New commits since handoff-24 (a049ff9..HEAD):

cd6371a tests: Phase 1 SQL SELECT integration tests
83e0ddc app: mode-threaded completion, overlay, and validity indicator
6369066 grammar: SQL SELECT end-to-end (ADR-0030 Phase 1)
c93f939 grammar: SQL expression grammar fragment (ADR-0031)
81793a3 docs: ADR-0031 — SQL expression grammar

All local; push asynchronously, not blocking.

§2. What was built — the shape now

Read docs/adr/0031-sql-expression-grammar.md for the grammar spec (Accepted) and docs/adr/0030-advanced-mode-sql- surface.md for the surrounding architecture.

2.1. ADR-0031: the SQL expression grammar

src/dsl/grammar/sql_expr.rs — a parallel fragment to grammar/expr.rs. Stratified ladder, one named static Node per precedence tier:

or_expr → and_expr → not_expr → predicate → additive →
  multiplicative → unary → primary

Predicate set: cmp_op / LIKE / BETWEEN / IN / IS NULL (with the ADR-0026 factoring — operand prefix matched once, infix NOT as an explicit branch). Primary covers literals, ( or_expr ), case_expr (searched + simple), and name_or_call — the identifier-prefix shared between column refs and function calls factored into one Ident followed by an Optional ( call_args ) tail (the same hazard-avoidance shape predicate_tail uses). count(*) and count(distinct …) both fall out of call_args.

No AST builder — every Phase-1 consumer (SELECT projection, WHERE) runs validated SQL as text per ADR-0030 §4/§6. The fragment's output is the matched-terminal MatchedPath (drives highlight / completion / hints) plus accept/reject. ADR-0031 §2 documents the rationale.

Drop-in: pub static SQL_EXPRESSION: Node = Subgrammar(&SQL_OR_EXPR);. SQL CommandNode shapes embed it the way DSL shapes embed expr::EXPRESSION. Recursion goes through Node::Subgrammar with ADR-0026's MAX_SUBGRAMMAR_DEPTH = 64 cap reused unchanged — no new walker capability.

2.2. Walker mode gate (ADR-0030 §2)

• WalkContext gains mode: Mode. Mode derives Default (= Simple, matching the app's startup mode).
• grammar::is_advanced_only keys the advanced-only entry-word set. Phase 1 has one entry: select.
• In walker::walk(), immediately after command_for_entry_word resolves the CommandNode, the gate short-circuits when ctx.mode == Simple && is_advanced_only(entry) — returning a WalkResult whose outcome is ValidationFailed { error: { key: "advanced_mode.sql_in_simple", … } }. The entry word still highlights as a keyword; the parse-error layer translates the catalog key into the "switch with mode advanced, or prefix the line with :" hint; the validity indicator goes ERROR.
• parser::parse_command_with_schema_in_mode (and the schemaless parse_command_in_mode) thread mode into WalkContext. Old parse_command* keep delegating with Mode::Advanced (most permissive — back-compat for tests + any caller that doesn't care about gating).

2.3. Builder signature sweep (ADR-0031 §2)

CommandNode.ast_builder is now fn(&MatchedPath, &str) -> Result<Command, ValidationError> — the walker forwards the parse source. build_select reads it to put the validated SQL text into Command::Select; the 21 pre-existing builders accept it as _source. The walker's call site flips from (&path) to (&path, source) in one place.

2.4. Command::Select + execution

• Command::Select { sql: String } — every exhaustive match Command updated (verb, target_table, build_translate_context, execute_command_typed, typing_surface_matrix's command_kind_label).
• grammar::data::SELECT — the CommandNode:

  projection ('*' | <expr>[ as <alias>][, ...])
    [ from <Table> ]
    [ where <sql_expr> ]
    [ order by <expr>[ asc|desc][, ...] ]
    [ limit <n> ]
    [ ; ]
  

  Expression slots reference sql_expr::SQL_OR_EXPR via Subgrammar. The FROM table-name slot carries a reject_internal_table validator that refuses any __rdbms_* reference at parse time (ADR-0030 §6).
• Worker: Request::RunSelect { sql, source, reply } + Database::run_select + do_run_select_request / do_run_select. The latter prepares the statement, collects rows into a DataResult with column_types: Vec<None> per ADR-0030 §6, and appends the literal source line to history.log so replay re-runs it (ADR-0030 §11).
• Runtime: execute_command_typed gains the Command::Select arm → database.run_select(sql, src) → CommandOutcome::Query → AppEvent::DslDataSucceeded → render_data_table. Reuses the existing show data rendering path end to end.

2.5. Dispatch unification + ambient mode threading

• App::submit is unified: both modes go through dispatch_dsl(&effective_input, effective_mode), which parses with the line's effective mode. The placeholder advanced-mode echo branch is gone (and the advanced_mode.not_implemented catalog entry is dead code now — left in place; remove on a passing future cleanup).
• EffectiveMode::as_mode() collapses the persistent / one-shot distinction into the plain Mode the walker reads.
• Walker exposes completion_probe_in_mode, expected_at_input_in_mode, input_verdict_in_mode. The empty-input / unknown-entry fallbacks in the first two filter the REGISTRY listing by is_advanced_only — Tab in simple mode does not offer select.
• Completion exposes candidates_at_cursor_in_mode / candidates_at_cursor_with_in_mode. Internally they route the parse_command completeness probe and the completion_probe / expected_at calls through the mode-aware variants.
• App::input_validity_verdict and the Tab handlers (start_or_complete_at / _last) pass Mode::Simple (validity indicator only fires in plain simple mode) and self.effective_mode().as_mode() (Tab handlers). input_render::render_input_runs and ambient_hint are invoked only when effective mode is plain Simple (ui.rs gates), so their internal calls hardcode Mode::Simple.

2.6. Catalog (ADR-0019)

• advanced_mode.sql_in_simple {command} — the walker gate hint.
• select.internal_table {table} — the __rdbms_* rejection.
• parse.usage.select — the parse-error usage template.

§3. Behaviour now (user-visible)

• Advanced-mode select runs: select 1, select * from t, select Name from Customers where Age > 18 order by Name limit 10 — all parse, dispatch as Command::Select, run via the worker, render as a data table.
• Simple-mode select does not run: the walker gates it, the dispatch path renders the catalog hint (advanced_mode.sql_in_simple), the validity indicator lights ERROR before submit, the live overlay marks the input as a definite error, Tab does not offer it as a completion.
• :select … in simple mode → one-shot Advanced, dispatches the SELECT; persistent mode unchanged.
• __rdbms_* in FROM → parse rejected with select.internal_table.
• history.log records every executed SELECT (literal line); replay re-runs them through the same dispatch path.

The data-table renderer treats SELECT results as a typeless view — column_types: Vec<None>. Pedagogically that means neutral alignment for everything; one specific UX note: bool columns SELECTed render as 0/1 rather than true/false. Phase 2 can enrich plain-column refs back to their source type.

§4. Autonomous decisions during execution

Per CLAUDE.md, these were judgement calls outside the original approach — they are flagged here so the next session can sanction or override:

1. FROM clause is optional. ADR-0030 Phase 1 says "single-table SELECT"; standard SQL also admits select 1 / select upper('x') (zero-table constant or function-call form). Included because it is the canonical learner probe; documented in code at the SELECT shape declaration.
2. Implicit projection aliasing (select a x) deliberately unsupported. A bare alias and the from keyword are structurally ambiguous; only select a as x is admitted.
3. help_id: None on the SELECT CommandNode. The help listing skips select for now — the help sql page is ADR-0030 Phase 6.
4. Some walker entries still default to internal Mode::Simple via WalkContext::new() — hint_mode_at_input*, hint_resolution_at_input, input_diagnostics, highlight_runs. Production paths that consume them are only invoked when effective mode is plain Simple (ui.rs gates), so the default is behaviourally correct. Full threading lands when advanced mode grows its own ambient assistance (Phase 2-ish).
5. SELECT result bool columns render as 0/1. All result column_types are None (ADR-0030 §6 — typeless view). Phase 2 can resolve plain column refs back to their schema type and recover the DSL show data rendering.

§5. What's next — ADR-0030 Phases 2–6

ADR-0030's plan continues. Each later phase is independently shippable; the two large grammar slices (Phase 2's full SELECT; Phase 3's SQL DML expression set if it grows) each warrant their own focused ADR (precedent: ADR-0026, ADR-0031). The order in the ADR is the suggested execution order; the user decides what to take next.

1. Phase 2 — SELECT (full). JOINs (inner / left / right), GROUP BY / HAVING, aggregate functions, subquery expressions ((SELECT …) as a primary; IN (SELECT …); EXISTS (…)), UNION / INTERSECT / EXCEPT, common table expressions (WITH), LIMIT … OFFSET, qualified column refs (t.c, alias.c). Its own focused ADR before authoring — same way ADR-0031 preceded the fragment. The expression grammar (ADR-0031) is shaped to receive the subquery and qualified-ref branches additively; no restructuring foreseen there.

2. Phase 3 — DML. SQL INSERT (single- and multi-row), UPDATE, DELETE. Execute as validated SQL (ADR-0030 §4) via the same Command::<X> { sql }-as-text pattern as Command::Select — the worker's "run validated DML + re- persist the table" request is new. Settle multi-row INSERT and shortid auto-fill on a SQL INSERT (open question, escalate to the user when reached).

3. Phase 4 — DDL. SQL CREATE/DROP/ALTER TABLE, CREATE/DROP INDEX. DDL routes through the typed Command executor (ADR-0030 §4) — the CommandNode's ast_builder translates SQL → Command::{CreateTable, AddColumn, …}, then the existing executor keeps metadata, the type vocabulary (ADR-0005) and STRICT intact. ADR-0030 §5 maps standard SQL type names (integer, varchar(255), numeric, …) onto the ten-type vocabulary. CHECK / DEFAULT expressions reuse the ADR-0031 fragment but their storage is text (ADR-0030 §11) — the Constraint::Check shape may need a text-carrying variant; escalate when reached. Table-rename (C1) may fall out here.

4. Phase 5 — DSL → SQL teaching echo. A DSL command run in advanced mode prints its equivalent SQL beneath the [ok] summary (ADR-0030 §10). It is a Command → SQL renderer; uses the OutputLine styled-runs mechanism (ADR-0028). App-level commands have no SQL form and are not echoed.

5. Phase 6 — Polish. help sql; engine-neutral error sweep (ADR-0030 §7); typing-surface / matrix coverage for the SQL surface; the DOC1 SQL-surface reference page.

§6. Seams for the next implementer

• sql_expr.rs is the grammar evolution point. Phase 2's subquery primary branch and qualified-column-ref tail are additive — a new Choice branch in primary, recursing through Subgrammar into the SELECT fragment, and a [ '.' identifier ] tail on name_or_call's ident. The ADR-0031 §7 OOS list calls these out by name.
• grammar/data.rs SELECT is the model for SQL DML/DDL CommandNodes (Phases 3-4) — same Subgrammar(&SQL_OR_EXPR) for expression slots, same reject_internal_table style validator on table-name slots, same _NODES / _SHAPE / CommandNode layout.
• Command::<X> { sql: String } pattern is the Phase-3 template for DML — build_<x>(_path, source) packages the validated text; the runtime arm sends it to a new "run + re-persist" worker request.
• WalkContext.mode is the gate; per-Choice-branch tagging (ADR-0030 §2 end state) is what arrives with the shared DSL/SQL entry words (create, insert, …) in Phase 3-4. The whole-command gating via is_advanced_only covers Phase 1 cleanly; per-branch tagging is a new mechanism on top — design it when the first shared entry word lands.
• Database::run_select is the read-only conn.prepare + collect-rows pattern. The Phase-3 DML "run validated SQL + re- persist the affected table" worker request reuses the conn.prepare half and adds re-persist (the existing do_query_data_request persistence call is a model).
• ast_builder now receives source: &str uniformly — SQL DML/DDL builders in Phases 3-4 use it freely; existing DSL builders ignore it as _source.
• parse_command*_in_mode is the right entry for any new schema/mode-sensitive call site; the unmoded parse_command* default to Mode::Advanced for back-compat. New production call sites should always pass an explicit mode.

§7. How to take over

1. Read this file, then docs/adr/0030-advanced-mode-sql- surface.md and docs/adr/0031-sql-expression-grammar.md. Then CLAUDE.md (working-style rules) and docs/ requirements.md (Q1/Q2 partially satisfied; Q4 ticked).
2. cargo test — 1260 passing, 0 failed, 1 ignored.
3. cargo clippy --all-targets -- -D warnings — clean.
4. Pick the next phase with the user. Phase 2 (full SELECT) is the natural next step and gets its own ADR before code, per ADR-0030 §3 — write the focused grammar ADR first (ADR-0026 / ADR-0031 are the model).
5. Escalate, do not guess, on the open per-phase questions ADR-0030 left — multi-row INSERT, shortid on SQL INSERT, the Constraint::Check text-carrying variant, any walker-taxonomy extension that changes the walker's contract.

§8. What else is open

Unchanged from handoff-24 §7 (prioritisation is a user decision): snapshot/undo U-series (ADR-0006 written, unbuilt); m:n convenience C4; modify-relationship C3a; the show family V5; rename-table C1 (may fall out of ADR-0030 Phase 4); friendly-error sweep H1; CI TT5; session-log / Markdown export V4.

Phase 1's specific deferrals (§4 of this file) — help_id: None on SELECT (Phase 6), the few walker entries still defaulting to internal Simple mode, bool rendering in SELECT results — are documented; the user decides whether to revisit any in Phase 2 or leave until Phase 6 polish.