rdbms-playground/docs/handoff/20260519-handoff-25.md

# Session handoff — 2026-05-19 (25)

Twenty-fifth handover. **Implementation session.** ADR-0030
Phase 1 ("Foundations + first `SELECT`") and ADR-0031 (the SQL
expression grammar) landed end to end: a SQL `select` typed in
advanced mode parses, runs against the database, and renders
through the existing data-table renderer; the same line in
simple mode lights up the precise "this is SQL" hint instead of
running, and Tab completion / live overlay / `[ERR]` indicator
agree.

## §1. State at handoff

**Branch:** `main`. Working tree clean.

**Tests:** **1260 passing, 0 failing, 1 ignored** (the
unchanged doctest). Up from the handoff-24 baseline of 1240 by
+20 — 13 sql-expression unit tests, 7 SQL-`SELECT` integration
tests. **Clippy:** clean under `-D warnings`.

**New commits since handoff-24** (`a049ff9..HEAD`):

```
cd6371a tests: Phase 1 SQL SELECT integration tests
83e0ddc app: mode-threaded completion, overlay, and validity indicator
6369066 grammar: SQL SELECT end-to-end (ADR-0030 Phase 1)
c93f939 grammar: SQL expression grammar fragment (ADR-0031)
81793a3 docs: ADR-0031 — SQL expression grammar
```

All local; push asynchronously, not blocking.

## §2. What was built — the shape now

Read **`docs/adr/0031-sql-expression-grammar.md`** for the
grammar spec (Accepted) and **`docs/adr/0030-advanced-mode-sql-
surface.md`** for the surrounding architecture.

### 2.1. ADR-0031: the SQL expression grammar

`src/dsl/grammar/sql_expr.rs` — a parallel fragment to
`grammar/expr.rs`. Stratified ladder, one named `static Node`
per precedence tier:

```
or_expr → and_expr → not_expr → predicate → additive →
  multiplicative → unary → primary
```

Predicate set: `cmp_op` / `LIKE` / `BETWEEN` / `IN` / `IS NULL`
(with the ADR-0026 factoring — operand prefix matched once,
infix `NOT` as an explicit branch). Primary covers literals,
`( or_expr )`, `case_expr` (searched + simple), and
`name_or_call` — the identifier-prefix shared between column
refs and function calls factored into one `Ident` followed by
an `Optional` `( call_args )` tail (the same hazard-avoidance
shape `predicate_tail` uses). `count(*)` and `count(distinct
…)` both fall out of `call_args`.

**No AST builder** — every Phase-1 consumer (SELECT projection,
WHERE) runs validated SQL as text per ADR-0030 §4/§6. The
fragment's output is the matched-terminal `MatchedPath` (drives
highlight / completion / hints) plus accept/reject. ADR-0031
§2 documents the rationale.

Drop-in: `pub static SQL_EXPRESSION: Node = Subgrammar(&SQL_OR_EXPR);`.
SQL `CommandNode` shapes embed it the way DSL shapes embed
`expr::EXPRESSION`. Recursion goes through `Node::Subgrammar`
with ADR-0026's `MAX_SUBGRAMMAR_DEPTH = 64` cap reused
unchanged — no new walker capability.

### 2.2. Walker mode gate (ADR-0030 §2)

- `WalkContext` gains `mode: Mode`. `Mode` derives `Default`
  (`= Simple`, matching the app's startup mode).
- `grammar::is_advanced_only` keys the advanced-only entry-word
  set. Phase 1 has one entry: `select`.
- In `walker::walk()`, immediately after `command_for_entry_word`
  resolves the `CommandNode`, the gate short-circuits when
  `ctx.mode == Simple && is_advanced_only(entry)` — returning a
  `WalkResult` whose outcome is `ValidationFailed { error: { key:
  "advanced_mode.sql_in_simple", … } }`. The entry word still
  highlights as a keyword; the parse-error layer translates the
  catalog key into the "switch with `mode advanced`, or prefix
  the line with `:`" hint; the validity indicator goes ERROR.
- `parser::parse_command_with_schema_in_mode` (and the
  schemaless `parse_command_in_mode`) thread `mode` into
  `WalkContext`. Old `parse_command*` keep delegating with
  `Mode::Advanced` (most permissive — back-compat for tests +
  any caller that doesn't care about gating).

### 2.3. Builder signature sweep (ADR-0031 §2)

`CommandNode.ast_builder` is now
`fn(&MatchedPath, &str) -> Result<Command, ValidationError>` —
the walker forwards the parse `source`. `build_select` reads it
to put the validated SQL text into `Command::Select`; the 21
pre-existing builders accept it as `_source`. The walker's call
site flips from `(&path)` to `(&path, source)` in one place.

### 2.4. `Command::Select` + execution

- `Command::Select { sql: String }` — every exhaustive
  `match Command` updated (`verb`, `target_table`,
  `build_translate_context`, `execute_command_typed`,
  `typing_surface_matrix`'s `command_kind_label`).
- `grammar::data::SELECT` — the `CommandNode`:
  ```
  projection ('*' | <expr>[ as <alias>][, ...])
    [ from <Table> ]
    [ where <sql_expr> ]
    [ order by <expr>[ asc|desc][, ...] ]
    [ limit <n> ]
    [ ; ]
  ```
  Expression slots reference `sql_expr::SQL_OR_EXPR` via
  `Subgrammar`. The `FROM` table-name slot carries a
  `reject_internal_table` validator that refuses any `__rdbms_*`
  reference at parse time (ADR-0030 §6).
- Worker: `Request::RunSelect { sql, source, reply }` +
  `Database::run_select` + `do_run_select_request` /
  `do_run_select`. The latter prepares the statement, collects
  rows into a `DataResult` with `column_types: Vec<None>` per
  ADR-0030 §6, and appends the literal source line to
  `history.log` so replay re-runs it (ADR-0030 §11).
- Runtime: `execute_command_typed` gains the `Command::Select`
  arm → `database.run_select(sql, src)` →
  `CommandOutcome::Query` → `AppEvent::DslDataSucceeded` →
  `render_data_table`. Reuses the existing `show data`
  rendering path end to end.

### 2.5. Dispatch unification + ambient mode threading

- `App::submit` is unified: both modes go through
  `dispatch_dsl(&effective_input, effective_mode)`, which parses
  with the line's effective mode. The placeholder advanced-mode
  echo branch is gone (and the `advanced_mode.not_implemented`
  catalog entry is dead code now — left in place; remove on a
  passing future cleanup).
- `EffectiveMode::as_mode()` collapses the persistent /
  one-shot distinction into the plain `Mode` the walker reads.
- Walker exposes `completion_probe_in_mode`,
  `expected_at_input_in_mode`, `input_verdict_in_mode`. The
  empty-input / unknown-entry fallbacks in the first two filter
  the `REGISTRY` listing by `is_advanced_only` — Tab in simple
  mode does not offer `select`.
- Completion exposes `candidates_at_cursor_in_mode` /
  `candidates_at_cursor_with_in_mode`. Internally they route the
  `parse_command` completeness probe and the `completion_probe`
  / `expected_at` calls through the mode-aware variants.
- `App::input_validity_verdict` and the Tab handlers
  (`start_or_complete_at` / `_last`) pass `Mode::Simple`
  (validity indicator only fires in plain simple mode) and
  `self.effective_mode().as_mode()` (Tab handlers).
  `input_render::render_input_runs` and `ambient_hint` are
  invoked only when effective mode is plain `Simple` (ui.rs
  gates), so their internal calls hardcode `Mode::Simple`.

### 2.6. Catalog (ADR-0019)

- `advanced_mode.sql_in_simple` `{command}` — the walker gate
  hint.
- `select.internal_table` `{table}` — the `__rdbms_*`
  rejection.
- `parse.usage.select` — the parse-error usage template.

## §3. Behaviour now (user-visible)

- **Advanced-mode `select`** runs: `select 1`, `select * from t`,
  `select Name from Customers where Age > 18 order by Name limit
  10` — all parse, dispatch as `Command::Select`, run via the
  worker, render as a data table.
- **Simple-mode `select`** does not run: the walker gates it,
  the dispatch path renders the catalog hint
  (`advanced_mode.sql_in_simple`), the validity indicator
  lights ERROR before submit, the live overlay marks the input
  as a definite error, Tab does not offer it as a completion.
- **`:select …`** in simple mode → one-shot Advanced, dispatches
  the SELECT; persistent mode unchanged.
- **`__rdbms_*` in `FROM`** → parse rejected with
  `select.internal_table`.
- **`history.log`** records every executed SELECT (literal
  line); `replay` re-runs them through the same dispatch path.

The data-table renderer treats SELECT results as a typeless
view — `column_types: Vec<None>`. Pedagogically that means
neutral alignment for everything; one specific UX note: bool
columns SELECTed render as `0`/`1` rather than `true`/`false`.
Phase 2 can enrich plain-column refs back to their source type.

## §4. Autonomous decisions during execution

Per CLAUDE.md, these were judgement calls outside the original
approach — they are flagged here so the next session can
sanction or override:

1. **`FROM` clause is optional.** ADR-0030 Phase 1 says
   "single-table SELECT"; standard SQL also admits `select 1` /
   `select upper('x')` (zero-table constant or function-call
   form). Included because it is the canonical learner probe;
   documented in code at the SELECT shape declaration.
2. **Implicit projection aliasing (`select a x`) deliberately
   unsupported.** A bare alias and the `from` keyword are
   structurally ambiguous; only `select a as x` is admitted.
3. **`help_id: None` on the SELECT `CommandNode`.** The `help`
   listing skips `select` for now — the `help sql` page is
   ADR-0030 Phase 6.
4. **Some walker entries still default to internal
   `Mode::Simple` via `WalkContext::new()`** —
   `hint_mode_at_input*`, `hint_resolution_at_input`,
   `input_diagnostics`, `highlight_runs`. Production paths that
   consume them are only invoked when effective mode is plain
   `Simple` (ui.rs gates), so the default is behaviourally
   correct. Full threading lands when advanced mode grows its
   own ambient assistance (Phase 2-ish).
5. **SELECT result bool columns render as `0`/`1`.** All result
   `column_types` are `None` (ADR-0030 §6 — typeless view).
   Phase 2 can resolve plain column refs back to their schema
   type and recover the DSL `show data` rendering.

## §5. What's next — ADR-0030 Phases 2–6

ADR-0030's plan continues. Each later phase is independently
shippable; the two large grammar slices (Phase 2's full
`SELECT`; Phase 3's SQL DML expression set if it grows) each
warrant their own focused ADR (precedent: ADR-0026, ADR-0031).
The order in the ADR is the suggested execution order; the
user decides what to take next.

1. **Phase 2 — `SELECT` (full).** `JOIN`s (inner / left /
   right), `GROUP BY` / `HAVING`, aggregate functions,
   subquery expressions (`(SELECT …)` as a primary;
   `IN (SELECT …)`; `EXISTS (…)`), `UNION` / `INTERSECT` /
   `EXCEPT`, common table expressions (`WITH`), `LIMIT … OFFSET`,
   qualified column refs (`t.c`, `alias.c`). Its own focused
   ADR before authoring — same way ADR-0031 preceded the
   fragment. The expression grammar (ADR-0031) is shaped to
   receive the subquery and qualified-ref branches additively;
   no restructuring foreseen there.

2. **Phase 3 — DML.** SQL `INSERT` (single- and multi-row),
   `UPDATE`, `DELETE`. Execute as validated SQL (ADR-0030 §4)
   via the same `Command::<X> { sql }`-as-text pattern as
   `Command::Select` — the worker's "run validated DML + re-
   persist the table" request is new. Settle multi-row
   `INSERT` and `shortid` auto-fill on a SQL `INSERT` (open
   question, escalate to the user when reached).

3. **Phase 4 — DDL.** SQL `CREATE`/`DROP`/`ALTER TABLE`,
   `CREATE`/`DROP INDEX`. DDL routes through the typed
   `Command` executor (ADR-0030 §4) — the `CommandNode`'s
   `ast_builder` translates SQL → `Command::{CreateTable,
   AddColumn, …}`, then the existing executor keeps metadata,
   the type vocabulary (ADR-0005) and `STRICT` intact. ADR-0030
   §5 maps standard SQL type names (`integer`, `varchar(255)`,
   `numeric`, …) onto the ten-type vocabulary. `CHECK` /
   `DEFAULT` expressions reuse the ADR-0031 fragment but their
   *storage* is text (ADR-0030 §11) — the `Constraint::Check`
   shape may need a text-carrying variant; escalate when
   reached. Table-rename (`C1`) may fall out here.

4. **Phase 5 — DSL → SQL teaching echo.** A DSL command run in
   advanced mode prints its equivalent SQL beneath the `[ok]`
   summary (ADR-0030 §10). It is a `Command` → SQL renderer;
   uses the `OutputLine` styled-runs mechanism (ADR-0028).
   App-level commands have no SQL form and are not echoed.

5. **Phase 6 — Polish.** `help sql`; engine-neutral error
   sweep (ADR-0030 §7); typing-surface / matrix coverage for
   the SQL surface; the `DOC1` SQL-surface reference page.

## §6. Seams for the next implementer

- **`sql_expr.rs` is the grammar evolution point.** Phase 2's
  subquery `primary` branch and qualified-column-ref tail are
  additive — a new `Choice` branch in `primary`, recursing
  through `Subgrammar` into the SELECT fragment, and a
  `[ '.' identifier ]` tail on `name_or_call`'s ident. The
  ADR-0031 §7 OOS list calls these out by name.
- **`grammar/data.rs` SELECT is the model for SQL DML/DDL
  CommandNodes** (Phases 3-4) — same `Subgrammar(&SQL_OR_EXPR)`
  for expression slots, same `reject_internal_table` style
  validator on table-name slots, same `_NODES` / `_SHAPE` /
  `CommandNode` layout.
- **`Command::<X> { sql: String }` pattern** is the Phase-3
  template for DML — `build_<x>(_path, source)` packages the
  validated text; the runtime arm sends it to a new
  "run + re-persist" worker request.
- **`WalkContext.mode` is the gate**; per-`Choice`-branch
  tagging (ADR-0030 §2 end state) is what arrives with the
  shared DSL/SQL entry words (`create`, `insert`, …) in Phase
  3-4. The whole-command gating via `is_advanced_only` covers
  Phase 1 cleanly; per-branch tagging is a new mechanism on top
  — design it when the first shared entry word lands.
- **`Database::run_select`** is the read-only `conn.prepare` +
  collect-rows pattern. The Phase-3 DML "run validated SQL + re-
  persist the affected table" worker request reuses the
  `conn.prepare` half and adds re-persist (the existing
  `do_query_data_request` persistence call is a model).
- **`ast_builder` now receives `source: &str`** uniformly — SQL
  DML/DDL builders in Phases 3-4 use it freely; existing DSL
  builders ignore it as `_source`.
- **`parse_command*_in_mode`** is the right entry for any new
  schema/mode-sensitive call site; the unmoded `parse_command*`
  default to `Mode::Advanced` for back-compat. New production
  call sites should always pass an explicit mode.

## §7. How to take over

1. **Read this file, then `docs/adr/0030-advanced-mode-sql-
   surface.md` and `docs/adr/0031-sql-expression-grammar.md`.**
   Then `CLAUDE.md` (working-style rules) and `docs/
   requirements.md` (Q1/Q2 partially satisfied; Q4 ticked).
2. **`cargo test`** — 1260 passing, 0 failed, 1 ignored.
3. **`cargo clippy --all-targets -- -D warnings`** — clean.
4. **Pick the next phase with the user.** Phase 2 (full SELECT)
   is the natural next step and gets its own ADR before code,
   per ADR-0030 §3 — write the focused grammar ADR first
   (ADR-0026 / ADR-0031 are the model).
5. **Escalate, do not guess**, on the open per-phase questions
   ADR-0030 left — multi-row `INSERT`, `shortid` on SQL
   `INSERT`, the `Constraint::Check` text-carrying variant, any
   walker-taxonomy extension that changes the walker's contract.

## §8. What else is open

Unchanged from handoff-24 §7 (prioritisation is a **user
decision**): snapshot/undo `U`-series (ADR-0006 written,
unbuilt); m:n convenience `C4`; modify-relationship `C3a`; the
`show` family `V5`; rename-table `C1` (may fall out of ADR-0030
Phase 4); friendly-error sweep `H1`; CI `TT5`; session-log /
Markdown export `V4`.

Phase 1's specific deferrals (§4 of this file) — `help_id: None`
on SELECT (Phase 6), the few walker entries still defaulting to
internal Simple mode, bool rendering in SELECT results — are
documented; the user decides whether to revisit any in Phase 2
or leave until Phase 6 polish.