rdbms-playground/docs/handoff/20260520-handoff-27.md

Session handoff — 2026-05-20 (27)

Twenty-seventh handover. Implementation session for ADR-0032 Phase 2. Sub-phases 2a, 2b (with §10.3 stage 2 deferred), 2c, 2d (with three diagnostic keys deferred), and 2f all shipped. Sub-phases 2e (completion + post-walk fixup) and 2g (verification sweep) remain.

§1. State at handoff

Branch: main. Working tree clean. 11 unpushed commits (be8a4f5..0c3847a plus the prior handoff-26 commit on top of 3db292c).

Tests: 1385 passing, 0 failing, 1 ignored (up from the 2026-05-19 handoff-26 baseline of 1260 / 0 / 1 — a net +125 tests).

Clippy: clean (last verified at end of 2f, commit 0c3847a).

Commits since handoff-26:

0c3847a db: column-origin type recovery in SELECT results (sub-phase 2f)
c5cf03b walker: SQL diagnostics — multi-binding scope, qualified refs, Phase-1 gap closure (sub-phase 2d)
a491df3 grammar: migrate Phase-1 SELECT to the ADR-0032 fragment (sub-phase 2c)
4ff054c walker: populate cte_bindings placeholders + projection_aliases (§10.3 stage 1 / §10.4)
b522d09 walker: populate from_scope table bindings (§10.1)
98a74b2 grammar: sql_expr additive extensions for §5/§6, CTE body rewires to ScopedSubgrammar
4f89106 walker: Node::ScopedSubgrammar variant + scope-frame stack (§10.2)
8d29335 grammar: SQL SELECT full statement fragment (Phase 2a)
e032f01 docs: ADR-0032 Amendment 1 — empirical scope of column-origin metadata

§2. What's done — the Phase-2 surface

A user typing in advanced mode can now write the full Phase-2 SELECT surface authored by ADR-0032 §§1–9:

• All five JOIN flavours (INNER / LEFT OUTER / RIGHT OUTER / FULL OUTER / CROSS) and bare JOIN.
• All four set ops (UNION / UNION ALL / INTERSECT / EXCEPT) in compound queries.
• CTEs — both WITH and WITH RECURSIVE, with optional (col-list). Dispatched via a new data::WITH CommandNode (entry word with is now in ADVANCED_ONLY_ENTRIES).
• GROUP BY / HAVING with arbitrary sql_expr keys.
• Qualified column refs (t.c / alias.c) — additive extension to sql_expr.rs's name_or_call (§5).
• Subquery expressions — scalar (SELECT …), IN (SELECT …), [NOT] EXISTS (SELECT …) — additive Choice branches in sql_expr.rs (§6), recursing through Node::ScopedSubgrammar(&SQL_SELECT_COMPOUND).
• LIMIT n [OFFSET m] with full sql_expr admission for both args (legacy LIMIT m, n rejected per §13 OOS-4).
• DISTINCT / ALL prefix.
• t.* qualified wildcard in projection.
• Bare-alias projection (SELECT a x FROM t — lifts Phase-1 §4.2) via per-position Node::Lookahead follow-set gating.
• OOS shapes reject: NATURAL JOIN, JOIN … USING, comma-FROM, derived tables in FROM, window functions (OVER (…)), VALUES row source, LATERAL (partial — the single-keyword form is admitted as a bare alias; the comma form rejects via OOS-3).

The Phase-1 data::SELECT shape migrated from data.rs into sql_select.rs's SQL_SELECT_TAIL (sub-phase 2c). Phase-1's 22 local grammar nodes plus the duplicate reject_internal_table validator are gone; the single source of truth for the SELECT shape is now sql_select.rs.

2.1. Walker capabilities

• Node::ScopedSubgrammar(&'static Node) (ADR-0032 §10.2) is the new node variant. Same recursion semantics as Subgrammar, plus a push/pop of ScopeFrame on the WalkContext::from_scope_stack. Shares subgrammar_depth with Subgrammar so the MAX_SUBGRAMMAR_DEPTH = 64 cap fires uniformly.
• WalkContext::from_scope_stack: Vec<ScopeFrame> — always non-empty (the bottom frame is the implicit top-level scope DSL paths use). Each ScopeFrame carries from_scope: Vec<TableBinding>, cte_bindings: Vec<CteBinding>, and projection_aliases: Vec<String>.
• Three new Node::Ident flags: writes_table_alias, writes_cte_name, writes_projection_alias. Together with the existing writes_table (now repurposed to push a TableBinding alongside its prior current_table write), they populate the top frame's accumulators during the walk. All 48 existing Ident sites carry the new flags as false (mechanical sed update; no behavioural change for DSL paths).
• current_table / current_table_columns stay as fields on WalkContext (additive approach — they're populated by writes_table alongside the from_scope push). Plan §2b's "make them derived helpers" became "keep them populated redundantly" — strictly safer, no behavioural drift for DSL paths, but a future-cleanup opportunity if the duplication proves a maintenance burden.

2.2. Diagnostics

Three diagnostic passes run on every successful parse (ADR-0032 §11.7):

1. schema_existence_diagnostics — extended for multi-binding scope (a pre-pass collects all table_name / cte_name / table_alias idents to sidestep the projection-before-FROM ordering problem). Emits diagnostic.unknown_table, diagnostic.unknown_column, diagnostic.unknown_qualifier, diagnostic.ambiguous_column, diagnostic.duplicate_cte (Plan §Open-2, user-approved).
2. sql_predicate_warnings — NEW. MatchedPath-walking variant of the predicate-warning pass, closes the Phase-1 carry-over gap (§11.6). Emits diagnostic.eq_null, diagnostic.like_numeric, diagnostic.type_mismatch on SQL WHERE / HAVING / JOIN ON / CASE / projection / ORDER BY slots. Scoped to bare column refs in <column> <op> <literal> form — qualified-ref and expression-operand cases stay un-flagged (safe false-negative posture).
3. DSL predicate_warnings (existing) — unchanged. Still walks the DSL Expr AST for DSL WHERE expressions.

2.3. Type recovery (sub-phase 2f)

Per ADR-0032 §12 + Amendment 1:

• Cargo.toml rusqlite features now include column_metadata.
• do_run_select calls resolve_select_column_types(conn, stmt) after prepare, queries __rdbms_playground_columns for each result column whose origin metadata is populated, and threads the type through to format_cell.
• ADR-0030 Phase-1 §4.5 (bool rendering as 0 / 1) is lifted for any bare-column reference whose origin the engine carries through — non-recursive CTE bodies, scalar subqueries, derived tables, set ops, JOINs all work for free per Amendment 1.

2.4. Catalog (ADR-0032 §11.5)

Six new diagnostic.* keys + eight new engine.* keys authored in src/friendly/strings/en-US.yaml and registered in src/friendly/keys.rs:

• diagnostic.ambiguous_column, diagnostic.compound_arity_mismatch, diagnostic.cte_arity_mismatch, diagnostic.duplicate_cte, diagnostic.projection_alias_misplaced, diagnostic.unknown_qualifier.
• engine.no_such_table, engine.no_such_column, engine.ambiguous_column, engine.aggregate_misuse, engine.group_by_required, engine.compound_arity_mismatch, engine.scalar_subquery_too_many_rows, engine.recursive_cte_malformed.

The engine.* keys are catalog entries only — the friendly-error layer reads them by key when reached, but no proactive enhancement of the layer's engine-message classification was pulled through here. That stays as a small follow-up.

§3. Deferrals — all user-approved or documented

§3.1. §10.3 stage 2 — CTE column-derivation harvest

Status: user-approved deferral (mid-session, 2026-05-20). Recorded in: docs/plans/20260520-adr-0032-phase-2.md §2b "Implementation status (2026-05-20)" block.

The placeholder CTE-binding push (stage 1) IS implemented — the CTE name is visible to the body (WITH RECURSIVE self-reference works) and to downstream CTE-name validators (2d's schema-existence pass accepts CTE-name table-source refs). What's missing is the six §10.3 derivation rules that populate the placeholder's columns at body-frame exit.

Knock-on effect: qualified-prefix completion past cte_alias.| returns an empty candidate list. Bare column refs into a CTE binding short-circuit to "accept silently" in 2d's schema-existence pass (we don't know the columns yet).

Folds into 2e (where qualified-prefix completion needs CTE columns) and any future arity-check pass that wants to compare declared (col-list) vs derived projection arity.

§3.2. Three 2d diagnostic keys deferred

projection_alias_misplaced, compound_arity_mismatch, and cte_arity_mismatch — catalog keys + strings are authored but the emitting logic is not implemented:

• projection_alias_misplaced needs clause-context detection (which clause an Ident sits in — the matched path is flat; some role-based recognition would work).
• compound_arity_mismatch needs per-leg projection counting as the compound is assembled.
• cte_arity_mismatch depends on §10.3 stage 2.

These were not explicitly user-approved as deferrals — they were noted in the 2d commit message as "deferred per the 2d-scoped DA review (documented as (TBD) in the cross-cut matrix for 2g)". The user should flag if these need to land before 2g signs off.

§3.3. Engine.* key wiring

Catalog entries authored; the friendly-error layer doesn't yet recognise the new engine messages and route them through the new keys. This is a small mechanical extension to the engine-error matcher; not blocking but worth folding in.

§4. What's next — sub-phases 2e and 2g

§4.1. Sub-phase 2e — completion + post-walk fixup

Per plan §2e:

• Qualified-prefix completion narrowing (§10.5): at the cursor in SELECT t.|, candidates should be t's columns (not the global column set). Requires:
  • The completion API gains a "prefix qualifier" hint that the walker passes when the cursor is immediately after Ident '.'.
  • The IdentSource::Columns completion path honours the hint when present, resolving against the top frame's from_scope binding for t.
• Post-walk projection-list fixup pass (§10.6): the projection-before-FROM problem. Collect projection Ident terminals during the walk; after walk completion, re-resolve each against the final from_scope; rewrite the highlight class and validity diagnostic.

Note: the §10.6 problem is partially sidestepped already in 2d — the schema-existence pass uses a two-pass approach that collects bindings first and then resolves. So projection-side identifiers don't get false-positive diagnostics from the existing pass. The 2e fixup-pass scope is more about getting the HIGHLIGHT class right at every keystroke (the typing experience) and supporting completion mid-typing. Worth re-reading §10.6 carefully with the 2d implementation in hand.

§4.2. Sub-phase 2g — integration sweep + verification

Per plan §2g:

• Fill in the cross-cut verification matrix (29 rows in docs/plans/20260520-adr-0032-phase-2.md). Many rows are satisfied by tests already authored in 2a–2f; the matrix needs to be walked and each row pointed at its test.
• Produce the final phase-exit verification report.
• DA final review.

§4.3. The §10.3 stage-2 harvest

If 2e wants real CTE-column-prefix completion, the harvest needs to land first. Six derivation rules per the ADR's table. Mechanism design: hook a callback into walk_scoped_subgrammar's on-exit path; when the frame being popped is a CTE body, walk its projection items and populate the placeholder binding in the outer frame. Projection item shape can be reconstructed from a projection_record accumulator on the frame (record the shape of each projection_item as the walker matches it — a new Node::ProjectionRecord wrapper, or a flag on Lookahead factories, would be the mechanism).

If the user is willing to defer CTE-prefix completion along with the harvest, 2e can ship without the harvest; qualified completion past cte.| returns nothing today and would continue to do so.

§5. Working tree and unpushed commits

e032f01 docs: ADR-0032 Amendment 1 — empirical scope of column-origin metadata
8d29335 grammar: SQL SELECT full statement fragment (Phase 2a)
4f89106 walker: Node::ScopedSubgrammar variant + scope-frame stack (§10.2)
98a74b2 grammar: sql_expr additive extensions for §5/§6, CTE body rewires to ScopedSubgrammar
b522d09 walker: populate from_scope table bindings (§10.1)
4ff054c walker: populate cte_bindings placeholders + projection_aliases (§10.3 stage 1 / §10.4)
a491df3 grammar: migrate Phase-1 SELECT to the ADR-0032 fragment (sub-phase 2c)
c5cf03b walker: SQL diagnostics — multi-binding scope, qualified refs, Phase-1 gap closure (sub-phase 2d)
0c3847a db: column-origin type recovery in SELECT results (sub-phase 2f)

Plus this handoff commit, once it lands.

§6. Seams for the next implementer

Where to look first:

• For 2e qualified-prefix completion: the existing IdentSource::Columns completion path in src/completion.rs (driven by WalkContext::current_table_columns). The qualified case needs a new "narrow to this binding's columns" pathway. The walker already records the sql_expr_qualified_ref role on the trailing ident in a t.c reference; the preceding Punct('.') and sql_expr_ident give the qualifier. Walker WalkContext could grow a pending_qualifier: Option<String> set when the walker enters a qualified-ref tail, cleared on successful match.

• For 2e post-walk fixup: the schema-existence pass in src/dsl/walker/mod.rs already does a pre-pass. The 2e fixup is about the HIGHLIGHT layer — rewriting per_byte_class entries for projection-list idents once the final from_scope is known. The walker's per_byte vec is the mutation target. Look at walker/highlight.rs for how byte-class runs are built today.

• For 2g verification matrix: walk row by row through docs/plans/20260520-adr-0032-phase-2.md's "Cross-cut verification matrix" table. Many rows already have tests; fill in the test location column. The handful that don't have explicit tests (the projection-before-FROM re-walk cycle, for example) may need new ones authored.

• For the §10.3 harvest: see §4.3 above. The cleanest mechanism is probably a new walker hook (a function pointer on a per-ScopedSubgrammar basis, or a usage flag on the Node::Ident variants that record projection items), but the design isn't fully nailed down. The implementer should think about whether to escalate the harvest design to the user before coding.

Code locations modified this session (in commit order):

• docs/adr/0032-sql-select-grammar.md — Amendment 1 added.
• docs/adr/README.md — index entry extended for Amendment 1.
• docs/plans/20260520-adr-0032-phase-2.md — §2b "Implementation status (2026-05-20)" block recording the §10.3 stage-2 deferral.
• src/dsl/grammar/mod.rs — Node::ScopedSubgrammar variant
  • three new Ident flags + data::WITH registered + ADVANCED_ONLY_ENTRIES extended with with.
• src/dsl/grammar/data.rs — ~150 lines of dead Phase-1 SELECT grammar removed; data::SELECT reshaped to Node::Subgrammar(&sql_select::SQL_SELECT_TAIL); data::WITH added.
• src/dsl/grammar/sql_select.rs — new file (sub-phase 2a), augmented through 2b and 2c.
• src/dsl/grammar/sql_expr.rs — additive §5 / §6 extensions.
• src/dsl/walker/context.rs — TableBinding, CteBinding, CteColumn, ScopeFrame; WalkContext::from_scope_stack.
• src/dsl/walker/driver.rs — Node::ScopedSubgrammar match arm; walk_scoped_subgrammar; writes_table_alias / writes_cte_name / writes_projection_alias handling in walk_ident.
• src/dsl/walker/mod.rs — multi-binding schema_existence_diagnostics rewrite, new sql_predicate_warnings pass, 16 new tests.
• src/friendly/keys.rs + src/friendly/strings/en-US.yaml — 6 new diagnostic.* + 8 new engine.* keys.
• Cargo.toml — column_metadata added to rusqlite features.
• src/db.rs — resolve_select_column_types helper; do_run_select threads per-column types through to format_cell.
• tests/sql_select.rs — 3 new SELECT type-recovery tests.

§7. How to take over

1. Read this file in full. Then docs/adr/0032-sql-select- grammar.md (and Amendment 1 — the empirical findings matter for 2f's design), docs/plans/20260520-adr-0032- phase-2.md, and CLAUDE.md.
2. cargo test — expect 1385 passing, 0 failed, 1 ignored.
3. cargo clippy --all-targets -- -D warnings — clean.
4. Pick the next sub-phase. 2e is substantial (qualified completion + post-walk fixup); 2g is the closing sweep (cross-cut matrix + verification report). Either is a reasonable next session.
5. Escalate before re-litigating deferrals. The three diagnostic deferrals in 2d (§3.2) are not user-approved — check whether they need to land before 2g signs off.
6. The §10.3 stage-2 harvest is the biggest open piece; if 2e wants real CTE-prefix completion, escalate the harvest design before coding.

§8. What's open beyond Phase 2

Unchanged from handoff-26 §8: snapshot/undo U-series; m:n convenience C4; modify-relationship C3a; rename-table C1; friendly-error sweep H1; CI TT5; session-log / Markdown export V4.

ADR-0030's later phases — DML (Phase 3), DDL (Phase 4), the DSL→SQL teaching echo (Phase 5), polish (Phase 6) — remain on the roadmap behind the Phase 2 sub-phases not yet completed.